[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:08:47 dist-test-slave-5l98 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:08:47 dist-test-slave-5l98 krb5kdc[395](info): set up 2 sockets
Sep 24 03:08:47 dist-test-slave-5l98 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 03:08:48.363125 388 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:08:48 dist-test-slave-5l98 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:08:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:08:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:08:48.566622 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34311
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:48.907932 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:48.908253 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:48.908342 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:48.915853 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:48.915977 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:48.916024 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:48.916117 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:48.916208 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:08:48.926352 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:08:48.928488 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:48.930040 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:48.944444 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:48.944387 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:48.944617 408 server_base.cc:1034] running on GCE node
W20260924 03:08:48.944387 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:48.945806 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:48.947366 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:48.948647 408 hybrid_clock.cc:648] HybridClock initialized: now 1790219328948449 us; error 174 us; skew 500 ppm
I20260924 03:08:48.949105 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:08:48 dist-test-slave-5l98 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:48.952883 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:08:48.954646 408 webserver.cc:460] Webserver started at http://127.0.97.62:35579/ using document root <none> and password file <none>
I20260924 03:08:48.955420 408 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:48.955551 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:48.955888 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:48.959470 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "7ad03a7123c64dabb283162f5e112d6b"
format_stamp: "Formatted at 2026-09-24 03:08:48 on dist-test-slave-5l98"
server_key: "125737c60d51bad28f04896dd97942ac"
server_key_iv: "8150efdd4516a2ee27a58a403a4f8906"
server_key_version: "encryptionkey@0"
I20260924 03:08:48.960309 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "7ad03a7123c64dabb283162f5e112d6b"
format_stamp: "Formatted at 2026-09-24 03:08:48 on dist-test-slave-5l98"
server_key: "125737c60d51bad28f04896dd97942ac"
server_key_iv: "8150efdd4516a2ee27a58a403a4f8906"
server_key_version: "encryptionkey@0"
I20260924 03:08:48.965955 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 03:08:48.969776 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:48.971369 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:08:48.971591 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "7ad03a7123c64dabb283162f5e112d6b"
format_stamp: "Formatted at 2026-09-24 03:08:48 on dist-test-slave-5l98"
server_key: "125737c60d51bad28f04896dd97942ac"
server_key_iv: "8150efdd4516a2ee27a58a403a4f8906"
server_key_version: "encryptionkey@0"
I20260924 03:08:48.971812 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:49.004333 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:49.010432 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:49.010774 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:49.033293 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34311
I20260924 03:08:49.033324 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34311 every 8 connection(s)
I20260924 03:08:49.034950 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:08:49.039523 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:49.045428 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 408
I20260924 03:08:49.045754 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:08:49.047816 388 external_mini_cluster.cc:1428] Setting key 387d1dec277b90f8a52ea347f3536886
I20260924 03:08:49.050010 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b: Bootstrap starting.
I20260924 03:08:49.054160 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:49.055678 475 log.cc:824] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:49.058499 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b: No bootstrap required, opened a new log
I20260924 03:08:49.067713 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad03a7123c64dabb283162f5e112d6b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34311 } }
I20260924 03:08:49.068194 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:49.068328 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad03a7123c64dabb283162f5e112d6b, State: Initialized, Role: FOLLOWER
I20260924 03:08:49.069063 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad03a7123c64dabb283162f5e112d6b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34311 } }
I20260924 03:08:49.069343 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:08:49.069510 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:08:49.069700 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:49.071836 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad03a7123c64dabb283162f5e112d6b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34311 } }
I20260924 03:08:49.072351 475 leader_election.cc:302] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ad03a7123c64dabb283162f5e112d6b; no voters:
I20260924 03:08:49.073035 475 leader_election.cc:288] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:08:49.073180 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:49.074798 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b [term 1 LEADER]: Becoming Leader. State: Replica: 7ad03a7123c64dabb283162f5e112d6b, State: Running, Role: LEADER
I20260924 03:08:49.075302 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad03a7123c64dabb283162f5e112d6b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34311 } }
I20260924 03:08:49.075662 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:49.078478 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ad03a7123c64dabb283162f5e112d6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad03a7123c64dabb283162f5e112d6b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34311 } } }
I20260924 03:08:49.078856 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b [sys.catalog]: This master's current role is: LEADER
Sep 24 03:08:49 dist-test-slave-5l98 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:08:49.085314 486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:08:49.086526 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ad03a7123c64dabb283162f5e112d6b. Latest consensus state: current_term: 1 leader_uuid: "7ad03a7123c64dabb283162f5e112d6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad03a7123c64dabb283162f5e112d6b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34311 } } }
I20260924 03:08:49.087419 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:49.093515 486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:08:49.100462 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:49.052128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54748 (local address 127.0.97.62:34311)
0924 03:08:49.053203 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:49.053220 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:49.054390 (+ 1170us) server_negotiation.cc:407] Connection header received
0924 03:08:49.067184 (+ 12794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:49.067212 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:49.067661 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:49.068159 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:49.071414 (+ 3255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:49.073102 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:49.076129 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:49.077415 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:49.077637 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:49.086457 (+ 8820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:49.086472 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:49.086496 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:49.086561 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:49.091695 (+ 5134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:49.092999 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:49.093010 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:49.093019 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:49.093290 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:49.093898 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:49.093903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:49.093905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:49.094648 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:49.097808 (+ 3160us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:49.098077 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 03:08:49.098567 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":458,"threads_started":1}
I20260924 03:08:49.109935 486 catalog_manager.cc:1347] Generated new cluster ID: 130d1d73f85149119496d35e87dfa995
I20260924 03:08:49.110072 486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:08:49.129832 486 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:08:49.132124 486 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:08:49.149513 486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ad03a7123c64dabb283162f5e112d6b: Generated new TSK 0
I20260924 03:08:49.150648 486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 03:08:49.201273 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34311
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:49.552277 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:49.552644 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:49.552802 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:49.560771 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:49.560887 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:49.561056 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:08:49.571228 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34311
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:08:49.572665 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:49.574016 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:49.588457 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:49.588577 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:49.588593 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:49.589957 501 server_base.cc:1034] running on GCE node
I20260924 03:08:49.590541 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:49.591454 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:49.592635 501 hybrid_clock.cc:648] HybridClock initialized: now 1790219329592586 us; error 42 us; skew 500 ppm
I20260924 03:08:49.592965 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:08:49 dist-test-slave-5l98 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:49.596195 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:08:49.598057 501 webserver.cc:460] Webserver started at http://127.0.97.1:46203/ using document root <none> and password file <none>
I20260924 03:08:49.598765 501 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:49.598910 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:49.599212 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:49.602135 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "1af2d30fe1d5454b8358a6bb1050d694"
format_stamp: "Formatted at 2026-09-24 03:08:49 on dist-test-slave-5l98"
server_key: "7c5f9de4570bacc1f2bf21f1bf2c8d95"
server_key_iv: "15c5dcada22993a97298a0736135960e"
server_key_version: "encryptionkey@0"
I20260924 03:08:49.603057 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1af2d30fe1d5454b8358a6bb1050d694"
format_stamp: "Formatted at 2026-09-24 03:08:49 on dist-test-slave-5l98"
server_key: "7c5f9de4570bacc1f2bf21f1bf2c8d95"
server_key_iv: "15c5dcada22993a97298a0736135960e"
server_key_version: "encryptionkey@0"
I20260924 03:08:49.608886 501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 03:08:49.612916 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:49.614348 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:08:49.614555 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "1af2d30fe1d5454b8358a6bb1050d694"
format_stamp: "Formatted at 2026-09-24 03:08:49 on dist-test-slave-5l98"
server_key: "7c5f9de4570bacc1f2bf21f1bf2c8d95"
server_key_iv: "15c5dcada22993a97298a0736135960e"
server_key_version: "encryptionkey@0"
I20260924 03:08:49.614866 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:49.630342 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:49.634831 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:49.635141 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:49.636157 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:49.637562 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:49.637676 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:49.637832 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:49.637920 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:49.677523 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43811
I20260924 03:08:49.677947 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43811 every 8 connection(s)
I20260924 03:08:49.678995 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:08:49.680855 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 501
I20260924 03:08:49.681075 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:08:49.681865 388 external_mini_cluster.cc:1428] Setting key 5675b7ce7d2186ebd8950bdb9506a7bf
Sep 24 03:08:49 dist-test-slave-5l98 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 03:08:49.701890 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:49.682113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55335 (local address 127.0.97.62:34311)
0924 03:08:49.682625 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:49.682631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:49.683809 (+ 1178us) server_negotiation.cc:407] Connection header received
0924 03:08:49.685853 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:49.685858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:49.685942 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:49.686290 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:49.687878 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:49.689136 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:49.690811 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:49.691263 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:49.691416 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:49.695295 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:49.695301 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:49.695305 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:49.695340 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:49.698523 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:49.699330 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:49.699337 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:49.699341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:49.699468 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:49.699935 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:49.699942 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:49.699945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:49.700225 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:49.700378 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:49.701283 (+ 905us) server_negotiation.cc:299] Negotiation successful
0924 03:08:49.701513 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":253,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 03:08:49.703223 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:49.682753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34311 (local address 127.0.97.1:55335)
0924 03:08:49.683642 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:49.683682 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:49.685536 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:49.686545 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:49.686554 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:49.687120 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:49.687706 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:49.687725 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:49.689353 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:49.689357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:49.690624 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:49.690637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:49.691505 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:49.691514 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:49.691803 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:49.692660 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:49.692687 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:49.695020 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:49.698708 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:49.698718 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:49.698739 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:49.699200 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:49.699581 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:49.699587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:49.699590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:49.699821 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:49.700352 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:49.700360 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:49.700895 (+ 535us) client_negotiation.cc:769] Sending connection context
0924 03:08:49.701202 (+ 307us) client_negotiation.cc:240] Negotiation successful
0924 03:08:49.701838 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":417,"threads_started":1}
I20260924 03:08:49.705134 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34311
I20260924 03:08:49.705432 629 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:49.706197 629 heartbeater.cc:511] Master 127.0.97.62:34311 requested a full tablet report, sending...
I20260924 03:08:49.708760 440 ts_manager.cc:194] Registered new tserver with Master: 1af2d30fe1d5454b8358a6bb1050d694 (127.0.97.1:43811)
I20260924 03:08:49.710234 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 03:08:49.729025 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34311
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:50.088135 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:50.088478 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:50.088747 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:50.097612 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:50.097743 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:50.097963 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:08:50.109403 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34311
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:08:50.110812 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:50.112304 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:50.127486 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:50.127431 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:50.127674 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:50.133780 635 server_base.cc:1034] running on GCE node
I20260924 03:08:50.134497 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:50.135604 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:50.136848 635 hybrid_clock.cc:648] HybridClock initialized: now 1790219330136789 us; error 49 us; skew 500 ppm
I20260924 03:08:50.137216 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:08:50 dist-test-slave-5l98 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:50.140805 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:08:50.142987 635 webserver.cc:460] Webserver started at http://127.0.97.2:45723/ using document root <none> and password file <none>
I20260924 03:08:50.143744 635 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:50.143891 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:50.144200 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:50.147179 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "017a507846154e36b6f5fe758c6226eb"
format_stamp: "Formatted at 2026-09-24 03:08:50 on dist-test-slave-5l98"
server_key: "fed55a8b89437d9ad0335553dc8b9fbf"
server_key_iv: "11ff942e783ab43f4b16fe26158a6302"
server_key_version: "encryptionkey@0"
I20260924 03:08:50.147957 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "017a507846154e36b6f5fe758c6226eb"
format_stamp: "Formatted at 2026-09-24 03:08:50 on dist-test-slave-5l98"
server_key: "fed55a8b89437d9ad0335553dc8b9fbf"
server_key_iv: "11ff942e783ab43f4b16fe26158a6302"
server_key_version: "encryptionkey@0"
I20260924 03:08:50.153383 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:08:50.157073 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:50.158407 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:08:50.158588 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "017a507846154e36b6f5fe758c6226eb"
format_stamp: "Formatted at 2026-09-24 03:08:50 on dist-test-slave-5l98"
server_key: "fed55a8b89437d9ad0335553dc8b9fbf"
server_key_iv: "11ff942e783ab43f4b16fe26158a6302"
server_key_version: "encryptionkey@0"
I20260924 03:08:50.158816 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:50.179244 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:50.184216 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:50.184561 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:50.185675 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:50.187180 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:50.187292 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:50.187446 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:50.187518 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:50.230888 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36301
I20260924 03:08:50.230935 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36301 every 8 connection(s)
I20260924 03:08:50.232225 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:08:50.238219 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 635
I20260924 03:08:50.238432 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:08:50.238946 388 external_mini_cluster.cc:1428] Setting key d4ff70a1a36957b0fa197f79f6a1b595
Sep 24 03:08:50 dist-test-slave-5l98 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 03:08:50.257170 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.234912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35493 (local address 127.0.97.62:34311)
0924 03:08:50.235373 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:50.235380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:50.236370 (+ 990us) server_negotiation.cc:407] Connection header received
0924 03:08:50.239322 (+ 2952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:50.239330 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:50.239418 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:50.239535 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:50.241736 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.242968 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.244874 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.245642 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.245792 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.250627 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:50.250634 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:50.250637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:50.250673 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:50.253705 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:50.254548 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:50.254556 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:50.254565 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:50.254693 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:50.255173 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:50.255181 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:50.255184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:50.255477 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:50.255665 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:50.256461 (+ 796us) server_negotiation.cc:299] Negotiation successful
0924 03:08:50.256754 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":221,"threads_started":1}
I20260924 03:08:50.258281 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.235474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34311 (local address 127.0.97.2:35493)
0924 03:08:50.236200 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:50.236238 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:50.238860 (+ 2622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:50.239923 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:50.239933 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:50.240678 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:50.241550 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:50.241570 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.243152 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.243156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:50.244702 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:50.244714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.246097 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.246109 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:50.246442 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.247212 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:50.247255 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:50.250330 (+ 3075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:50.253898 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:50.253909 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:50.253932 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:50.254401 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:50.254820 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:50.254826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:50.254830 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:50.255041 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:50.255620 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:50.255629 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:50.256050 (+ 421us) client_negotiation.cc:769] Sending connection context
0924 03:08:50.256343 (+ 293us) client_negotiation.cc:240] Negotiation successful
0924 03:08:50.257021 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":297,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 03:08:50.260104 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34311
I20260924 03:08:50.260620 763 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:50.261493 763 heartbeater.cc:511] Master 127.0.97.62:34311 requested a full tablet report, sending...
I20260924 03:08:50.263279 440 ts_manager.cc:194] Registered new tserver with Master: 017a507846154e36b6f5fe758c6226eb (127.0.97.2:36301)
I20260924 03:08:50.264475 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 03:08:50.284741 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34311
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:50.633884 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:50.634171 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:50.634318 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:50.642912 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:50.643038 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:50.643234 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:08:50.653294 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34311
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:08:50.654625 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:50.655985 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:50.669679 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:50.669651 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:50.669770 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:50.670588 769 server_base.cc:1034] running on GCE node
I20260924 03:08:50.671300 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:50.672338 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:50.673571 769 hybrid_clock.cc:648] HybridClock initialized: now 1790219330673538 us; error 54 us; skew 500 ppm
I20260924 03:08:50.673902 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:08:50 dist-test-slave-5l98 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:50.678006 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:08:50.682861 769 webserver.cc:460] Webserver started at http://127.0.97.3:36855/ using document root <none> and password file <none>
I20260924 03:08:50.683724 769 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:50.683868 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:50.684216 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:50.687085 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "6b7fe5bcfe25468cb9676b625f79a266"
format_stamp: "Formatted at 2026-09-24 03:08:50 on dist-test-slave-5l98"
server_key: "1b79494c065f749ff277eecc578499d9"
server_key_iv: "266fb647cba1a3508d4c3ff24f328330"
server_key_version: "encryptionkey@0"
I20260924 03:08:50.687817 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6b7fe5bcfe25468cb9676b625f79a266"
format_stamp: "Formatted at 2026-09-24 03:08:50 on dist-test-slave-5l98"
server_key: "1b79494c065f749ff277eecc578499d9"
server_key_iv: "266fb647cba1a3508d4c3ff24f328330"
server_key_version: "encryptionkey@0"
I20260924 03:08:50.693287 769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 03:08:50.697053 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:50.698555 769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:08:50.698763 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "6b7fe5bcfe25468cb9676b625f79a266"
format_stamp: "Formatted at 2026-09-24 03:08:50 on dist-test-slave-5l98"
server_key: "1b79494c065f749ff277eecc578499d9"
server_key_iv: "266fb647cba1a3508d4c3ff24f328330"
server_key_version: "encryptionkey@0"
I20260924 03:08:50.698964 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:50.712918 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:50.713375 629 heartbeater.cc:503] Master 127.0.97.62:34311 was elected leader, sending a full tablet report...
I20260924 03:08:50.717975 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:50.718310 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:50.719456 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:50.720969 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:50.721076 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:50.721203 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:50.721292 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:50.764274 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44389
I20260924 03:08:50.764367 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44389 every 8 connection(s)
I20260924 03:08:50.765621 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:08:50.773512 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 769
I20260924 03:08:50.773710 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:08:50.774134 388 external_mini_cluster.cc:1428] Setting key 315363662c755eb5d85dc4e67daeb3f3
Sep 24 03:08:50 dist-test-slave-5l98 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:08:50.789803 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.768814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47525 (local address 127.0.97.62:34311)
0924 03:08:50.769275 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:50.769280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:50.770302 (+ 1022us) server_negotiation.cc:407] Connection header received
0924 03:08:50.772307 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:50.772313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:50.772416 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:50.772563 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:50.774322 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.775087 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.777299 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.777875 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.777988 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.781799 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:50.781810 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:50.781822 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:50.781884 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:50.785870 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:50.786722 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:50.786727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:50.786729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:50.786947 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:50.787430 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:50.787432 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:50.787434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:50.787690 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:50.787808 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:50.788629 (+ 821us) server_negotiation.cc:299] Negotiation successful
0924 03:08:50.788986 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":234,"threads_started":1}
I20260924 03:08:50.790571 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.769423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34311 (local address 127.0.97.3:47525)
0924 03:08:50.770132 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:50.770174 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:50.772032 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:50.772790 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:50.772798 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:50.773380 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:50.774120 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:50.774139 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.775293 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.775299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:50.777049 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:50.777063 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.778135 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.778141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:50.778352 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.779110 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:50.779133 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:50.781517 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:50.786031 (+ 4514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:50.786039 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:50.786053 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:50.786564 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:50.787103 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:50.787107 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:50.787109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:50.787334 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:50.787808 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:50.787815 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:50.788150 (+ 335us) client_negotiation.cc:769] Sending connection context
0924 03:08:50.788462 (+ 312us) client_negotiation.cc:240] Negotiation successful
0924 03:08:50.789063 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":272,"threads_started":1}
I20260924 03:08:50.792084 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34311
I20260924 03:08:50.792410 897 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:50.793346 897 heartbeater.cc:511] Master 127.0.97.62:34311 requested a full tablet report, sending...
I20260924 03:08:50.794997 440 ts_manager.cc:194] Registered new tserver with Master: 6b7fe5bcfe25468cb9676b625f79a266 (127.0.97.3:44389)
I20260924 03:08:50.796670 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47525
I20260924 03:08:50.804034 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:08:50 dist-test-slave-5l98 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219330, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:08:50 dist-test-slave-5l98 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219330, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:08:50.866012 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.848324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52362 (local address 127.0.97.1:43811)
0924 03:08:50.849354 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:50.849367 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:50.849417 (+ 50us) server_negotiation.cc:407] Connection header received
0924 03:08:50.849496 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:50.849508 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:50.849845 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:50.850047 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:50.851713 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.853059 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.854469 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.855090 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.855204 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.859770 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:50.859785 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:50.859799 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:50.859846 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:50.862843 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:50.863785 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:50.863794 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:50.863797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:50.864041 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:50.864473 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:50.864476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:50.864478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:50.864937 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:50.865191 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:50.865482 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 03:08:50.865686 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829,"thread_start_us":678,"threads_started":1}
W20260924 03:08:50.867439 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52362
Sep 24 03:08:50 dist-test-slave-5l98 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:08:50 dist-test-slave-5l98 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:08:50.931773 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.916796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52368 (local address 127.0.97.1:43811)
0924 03:08:50.917058 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:50.917068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:50.917388 (+ 320us) server_negotiation.cc:407] Connection header received
0924 03:08:50.917585 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:50.917590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:50.917658 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:50.917758 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:50.919506 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.920283 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.921656 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.922125 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.922227 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.926434 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:50.926439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:50.926442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:50.926475 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:50.929184 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:50.929824 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:50.929831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:50.929839 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:50.929958 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:50.930326 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:50.930331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:50.930334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:50.930578 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:50.930815 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:50.931126 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 03:08:50.931334 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 03:08:50.933377 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 501
I20260924 03:08:50.947556 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.934441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52384 (local address 127.0.97.1:43811)
0924 03:08:50.934649 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:50.934655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:50.934925 (+ 270us) server_negotiation.cc:407] Connection header received
0924 03:08:50.935132 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:50.935137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:50.935227 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:50.935342 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:50.937042 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.938243 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.939507 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.940004 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.940101 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.942411 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:50.942417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:50.942421 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:50.942450 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:50.945101 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:50.945694 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:50.945698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:50.945700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:50.945927 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:50.946241 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:50.946244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:50.946246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:50.946524 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:50.946694 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:50.946942 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 03:08:50.947086 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:08:50.948264 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:08:51.033418 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 501
I20260924 03:08:51.051697 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 635
Sep 24 03:08:51 dist-test-slave-5l98 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 03:08:51.069844 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.052747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60226 (local address 127.0.97.2:36301)
0924 03:08:51.053292 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:51.053300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:51.053324 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:08:51.053397 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:51.053403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:51.053606 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:51.053782 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:51.055003 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:51.056025 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:51.057426 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:51.058018 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:51.058125 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:51.062905 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:51.062920 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:51.062948 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:51.062995 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:51.066646 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:51.067508 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:51.067515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:51.067518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:51.067648 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:51.068048 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:51.068052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:51.068055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:51.068617 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:51.068911 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:51.069231 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 03:08:51.069434 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":281,"threads_started":1}
I20260924 03:08:51.071317 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:08:51.157598 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 635
I20260924 03:08:51.180174 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 769
Sep 24 03:08:51 dist-test-slave-5l98 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 03:08:51.198388 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.181368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54630 (local address 127.0.97.3:44389)
0924 03:08:51.182003 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:51.182011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:51.182040 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:08:51.182098 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:51.182105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:51.182342 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:51.182543 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:51.183904 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:51.185312 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:51.186915 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:51.187510 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:51.187683 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:51.192200 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:51.192215 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:51.192231 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:51.192270 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:51.195400 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:51.196164 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:51.196173 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:51.196176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:51.196372 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:51.196790 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:51.196793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:51.196795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:51.197145 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:51.197389 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:51.197819 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 03:08:51.198104 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":262,"threads_started":1}
I20260924 03:08:51.199766 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:08:51.285437 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 769
I20260924 03:08:51.301990 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 408
I20260924 03:08:51.303452 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:08:51.406448 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 408
2026-09-24T03:08:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4205 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:08:51 dist-test-slave-5l98 krb5kdc[919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:08:51 dist-test-slave-5l98 krb5kdc[919](info): set up 2 sockets
Sep 24 03:08:51 dist-test-slave-5l98 krb5kdc[919](info): commencing operation
krb5kdc: starting...
W20260924 03:08:52.508841 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.007s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:08:52 dist-test-slave-5l98 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:08:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:08:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:08:52.697677 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34425
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34425
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:53.023151 932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:53.023496 932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:53.023584 932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:53.030757 932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:53.030886 932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:53.030933 932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:53.030982 932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:53.031025 932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:08:53.040791 932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:08:53.042101 932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:53.043327 932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:53.055521 940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:53.055521 937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:53.055588 938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:53.056651 932 server_base.cc:1034] running on GCE node
I20260924 03:08:53.057502 932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:53.058892 932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:53.060094 932 hybrid_clock.cc:648] HybridClock initialized: now 1790219333060018 us; error 63 us; skew 500 ppm
I20260924 03:08:53.060459 932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:08:53 dist-test-slave-5l98 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:53.063803 932 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:08:53.065457 932 webserver.cc:460] Webserver started at http://127.0.97.62:39849/ using document root <none> and password file <none>
I20260924 03:08:53.066114 932 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:53.066236 932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:53.066509 932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:53.069494 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "eb3aa4af2d7f47acbf77513572c0a472"
format_stamp: "Formatted at 2026-09-24 03:08:53 on dist-test-slave-5l98"
server_key: "f5421ec9dd2e7b0ce046147bd48598e5"
server_key_iv: "6dd8f1a802dd4d50f975b13208df12e7"
server_key_version: "encryptionkey@0"
I20260924 03:08:53.070220 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "eb3aa4af2d7f47acbf77513572c0a472"
format_stamp: "Formatted at 2026-09-24 03:08:53 on dist-test-slave-5l98"
server_key: "f5421ec9dd2e7b0ce046147bd48598e5"
server_key_iv: "6dd8f1a802dd4d50f975b13208df12e7"
server_key_version: "encryptionkey@0"
I20260924 03:08:53.075398 932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:08:53.078974 947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:53.080411 932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 03:08:53.080680 932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "eb3aa4af2d7f47acbf77513572c0a472"
format_stamp: "Formatted at 2026-09-24 03:08:53 on dist-test-slave-5l98"
server_key: "f5421ec9dd2e7b0ce046147bd48598e5"
server_key_iv: "6dd8f1a802dd4d50f975b13208df12e7"
server_key_version: "encryptionkey@0"
I20260924 03:08:53.080964 932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:53.107169 932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:53.111850 932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:53.112164 932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:53.131218 932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34425
I20260924 03:08:53.131248 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34425 every 8 connection(s)
I20260924 03:08:53.132612 932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:08:53.135805 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 932
I20260924 03:08:53.136034 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:08:53.136701 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:53.136878 388 external_mini_cluster.cc:1428] Setting key df6834e3f7045126ca6c3e51feafb2cf
I20260924 03:08:53.149386 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472: Bootstrap starting.
Sep 24 03:08:53 dist-test-slave-5l98 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:08:53.152697 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:53.153771 999 log.cc:824] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:53.156667 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472: No bootstrap required, opened a new log
I20260924 03:08:53.159123 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:53.138960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46028 (local address 127.0.97.62:34425)
0924 03:08:53.139822 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:53.139838 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:53.139880 (+ 42us) server_negotiation.cc:407] Connection header received
0924 03:08:53.141633 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:53.141655 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:53.142050 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:53.142455 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:53.143877 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:53.145217 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:53.146469 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:53.147177 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:53.147384 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:53.151619 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:53.151627 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:53.151643 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:53.151679 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:53.154971 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:53.155683 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:53.155690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:53.155696 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:53.155862 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:53.156233 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:53.156239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:53.156244 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:53.156867 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:53.157171 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:53.157660 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 03:08:53.158071 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":372,"threads_started":1}
I20260924 03:08:53.164829 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3aa4af2d7f47acbf77513572c0a472" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34425 } }
I20260924 03:08:53.165247 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:53.165361 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3aa4af2d7f47acbf77513572c0a472, State: Initialized, Role: FOLLOWER
I20260924 03:08:53.165930 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3aa4af2d7f47acbf77513572c0a472" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34425 } }
I20260924 03:08:53.166189 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:08:53.166342 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:08:53.166522 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:53.168747 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3aa4af2d7f47acbf77513572c0a472" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34425 } }
I20260924 03:08:53.169251 999 leader_election.cc:302] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb3aa4af2d7f47acbf77513572c0a472; no voters:
I20260924 03:08:53.169808 999 leader_election.cc:288] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:08:53.169950 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:53.171029 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472 [term 1 LEADER]: Becoming Leader. State: Replica: eb3aa4af2d7f47acbf77513572c0a472, State: Running, Role: LEADER
I20260924 03:08:53.171638 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3aa4af2d7f47acbf77513572c0a472" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34425 } }
I20260924 03:08:53.172616 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:53.174685 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb3aa4af2d7f47acbf77513572c0a472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3aa4af2d7f47acbf77513572c0a472" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34425 } } }
I20260924 03:08:53.174978 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb3aa4af2d7f47acbf77513572c0a472. Latest consensus state: current_term: 1 leader_uuid: "eb3aa4af2d7f47acbf77513572c0a472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3aa4af2d7f47acbf77513572c0a472" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34425 } } }
I20260924 03:08:53.175246 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472 [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:53.175297 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472 [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:53.181454 1011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:08:53.187083 1011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:08:53.196635 1011 catalog_manager.cc:1347] Generated new cluster ID: 2dd1ade3764c41a6a517301d5159bf05
I20260924 03:08:53.196798 1011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:08:53.219898 1011 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:08:53.222031 1011 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:08:53.234067 1011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb3aa4af2d7f47acbf77513572c0a472: Generated new TSK 0
I20260924 03:08:53.235215 1011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 03:08:53.286782 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34425
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:53.785467 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:53.785753 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:53.785956 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:53.793452 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:53.793556 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:53.793766 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:08:53.803913 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34425
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:08:53.806099 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:53.808264 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:53.827709 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:53.827875 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:53.828228 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:53.828785 1025 server_base.cc:1034] running on GCE node
I20260924 03:08:53.829437 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:53.830533 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:53.831776 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790219333831729 us; error 35 us; skew 500 ppm
I20260924 03:08:53.832147 1025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:08:53 dist-test-slave-5l98 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:53.835736 1025 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:08:53.837381 1025 webserver.cc:460] Webserver started at http://127.0.97.1:42115/ using document root <none> and password file <none>
I20260924 03:08:53.838097 1025 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:53.838253 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:53.838574 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:53.841778 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "4938313f3b3d4a3eb5ace185bcad3395"
format_stamp: "Formatted at 2026-09-24 03:08:53 on dist-test-slave-5l98"
server_key: "3d56babe4acd8e765b520ca6cf60637d"
server_key_iv: "b469c3c55682e0f42af98b405a5bd90d"
server_key_version: "encryptionkey@0"
I20260924 03:08:53.842592 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4938313f3b3d4a3eb5ace185bcad3395"
format_stamp: "Formatted at 2026-09-24 03:08:53 on dist-test-slave-5l98"
server_key: "3d56babe4acd8e765b520ca6cf60637d"
server_key_iv: "b469c3c55682e0f42af98b405a5bd90d"
server_key_version: "encryptionkey@0"
I20260924 03:08:53.848129 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:08:53.852097 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:53.853717 1025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260924 03:08:53.853878 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "4938313f3b3d4a3eb5ace185bcad3395"
format_stamp: "Formatted at 2026-09-24 03:08:53 on dist-test-slave-5l98"
server_key: "3d56babe4acd8e765b520ca6cf60637d"
server_key_iv: "b469c3c55682e0f42af98b405a5bd90d"
server_key_version: "encryptionkey@0"
I20260924 03:08:53.854110 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:53.877048 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:53.881850 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:53.882180 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:53.883237 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:53.884725 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:53.884800 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:53.884954 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:53.885033 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:53.917438 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37693
I20260924 03:08:53.917438 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37693 every 8 connection(s)
I20260924 03:08:53.918802 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:08:53.928064 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 1025
I20260924 03:08:53.928340 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:08:53.929292 388 external_mini_cluster.cc:1428] Setting key 177c909460e7a45c7178268ce54a4957
Sep 24 03:08:53 dist-test-slave-5l98 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:08:53.942728 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:53.921561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45647 (local address 127.0.97.62:34425)
0924 03:08:53.922093 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:53.922098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:53.923087 (+ 989us) server_negotiation.cc:407] Connection header received
0924 03:08:53.924966 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:53.924971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:53.925043 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:53.925149 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:53.926912 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:53.927691 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:53.930356 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:53.930848 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:53.930958 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:53.935419 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:53.935425 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:53.935428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:53.935461 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:53.939083 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:53.939991 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:53.940000 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:53.940006 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:53.940160 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:53.940677 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:53.940684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:53.940690 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:53.941056 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:53.941202 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:53.941961 (+ 759us) server_negotiation.cc:299] Negotiation successful
0924 03:08:53.942228 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":314,"threads_started":1}
I20260924 03:08:53.943536 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:53.922127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34425 (local address 127.0.97.1:45647)
0924 03:08:53.922920 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:53.922964 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:53.924672 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:53.925383 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:53.925402 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:53.926071 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:53.926649 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:53.926666 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:53.927966 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:53.927973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:53.930194 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:53.930206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:53.931105 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:53.931113 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:53.931386 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:53.932165 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:53.932205 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:53.935138 (+ 2933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:53.939309 (+ 4171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:53.939317 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:53.939333 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:53.939843 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:53.940286 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:53.940290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:53.940292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:53.940484 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:53.941196 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:53.941203 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:53.941554 (+ 351us) client_negotiation.cc:769] Sending connection context
0924 03:08:53.941799 (+ 245us) client_negotiation.cc:240] Negotiation successful
0924 03:08:53.942259 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":277,"threads_started":1}
I20260924 03:08:53.945331 1153 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34425
I20260924 03:08:53.945652 1153 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:53.946409 1153 heartbeater.cc:511] Master 127.0.97.62:34425 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 03:08:53.949225 964 ts_manager.cc:194] Registered new tserver with Master: 4938313f3b3d4a3eb5ace185bcad3395 (127.0.97.1:37693)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 03:08:53.951294 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 03:08:53.977135 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34425
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:54.308516 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:54.308830 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:54.308969 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:54.316264 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:54.316385 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:54.316617 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:08:54.326522 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34425
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:08:54.327880 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:54.329288 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:54.342545 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:54.342571 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:54.342530 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:54.343225 1159 server_base.cc:1034] running on GCE node
I20260924 03:08:54.344014 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:54.345147 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:54.346438 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790219334346362 us; error 53 us; skew 500 ppm
I20260924 03:08:54.346895 1159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:08:54 dist-test-slave-5l98 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:54.350579 1159 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:08:54.352262 1159 webserver.cc:460] Webserver started at http://127.0.97.2:46719/ using document root <none> and password file <none>
I20260924 03:08:54.353032 1159 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:54.353135 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:54.353399 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:54.356401 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "1ef8dd8eedaa4c7e9ab9aec60c17cc70"
format_stamp: "Formatted at 2026-09-24 03:08:54 on dist-test-slave-5l98"
server_key: "e29a6c91c2b979bdfe695fe8a9d21392"
server_key_iv: "b6cacce7b5dff08d3402b97cd4884294"
server_key_version: "encryptionkey@0"
I20260924 03:08:54.357254 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1ef8dd8eedaa4c7e9ab9aec60c17cc70"
format_stamp: "Formatted at 2026-09-24 03:08:54 on dist-test-slave-5l98"
server_key: "e29a6c91c2b979bdfe695fe8a9d21392"
server_key_iv: "b6cacce7b5dff08d3402b97cd4884294"
server_key_version: "encryptionkey@0"
I20260924 03:08:54.362915 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:08:54.366658 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:54.368257 1159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:08:54.368445 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "1ef8dd8eedaa4c7e9ab9aec60c17cc70"
format_stamp: "Formatted at 2026-09-24 03:08:54 on dist-test-slave-5l98"
server_key: "e29a6c91c2b979bdfe695fe8a9d21392"
server_key_iv: "b6cacce7b5dff08d3402b97cd4884294"
server_key_version: "encryptionkey@0"
I20260924 03:08:54.368825 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:54.394717 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:54.399466 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:54.399849 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:54.400939 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:54.402350 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:54.402424 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:54.402593 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:54.402644 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:54.436801 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40221
I20260924 03:08:54.436846 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40221 every 8 connection(s)
I20260924 03:08:54.438153 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:08:54.446105 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 1159
I20260924 03:08:54.446312 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:08:54.446734 388 external_mini_cluster.cc:1428] Setting key c8b046bbe8935397d44375c283f839b8
Sep 24 03:08:54 dist-test-slave-5l98 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:08:54.461606 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:54.440753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57691 (local address 127.0.97.62:34425)
0924 03:08:54.440930 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:54.440935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:54.442156 (+ 1221us) server_negotiation.cc:407] Connection header received
0924 03:08:54.443982 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:54.443988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:54.444083 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:54.444195 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:54.445828 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:54.446621 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:54.448354 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:54.448863 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:54.449008 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:54.453275 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:54.453285 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:54.453291 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:54.453341 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:54.457577 (+ 4236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:54.458441 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:54.458450 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:54.458457 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:54.458615 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:54.459208 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:54.459217 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:54.459224 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:54.459679 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:54.459844 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:54.460812 (+ 968us) server_negotiation.cc:299] Negotiation successful
0924 03:08:54.461093 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 03:08:54.462255 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:54.441285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34425 (local address 127.0.97.2:57691)
0924 03:08:54.441992 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:54.442036 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:54.443704 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:54.444464 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:54.444478 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:54.445069 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:54.445625 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:54.445642 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:54.446779 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:54.446783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:54.448193 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:54.448205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:54.449114 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:54.449123 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:54.449342 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:54.450120 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:54.450144 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:54.453012 (+ 2868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:54.457748 (+ 4736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:54.457756 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:54.457770 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:54.458285 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:54.458793 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:54.458797 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:54.458799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:54.458993 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:54.459912 (+ 919us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:54.459918 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:54.460270 (+ 352us) client_negotiation.cc:769] Sending connection context
0924 03:08:54.460671 (+ 401us) client_negotiation.cc:240] Negotiation successful
0924 03:08:54.461089 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":273,"threads_started":1}
I20260924 03:08:54.463909 1287 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34425
I20260924 03:08:54.464213 1287 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:54.464994 1287 heartbeater.cc:511] Master 127.0.97.62:34425 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 03:08:54.466545 964 ts_manager.cc:194] Registered new tserver with Master: 1ef8dd8eedaa4c7e9ab9aec60c17cc70 (127.0.97.2:40221)
I20260924 03:08:54.467347 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 03:08:54.493991 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34425
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:54.821498 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:54.821777 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:54.821941 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:54.829258 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:54.829380 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:54.829584 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:08:54.840018 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34425
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:08:54.841425 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:54.842804 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:54.856624 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:54.856596 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:54.856709 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:54.857484 1292 server_base.cc:1034] running on GCE node
I20260924 03:08:54.858089 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:54.859181 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:54.860442 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790219334860351 us; error 62 us; skew 500 ppm
I20260924 03:08:54.860852 1292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:08:54 dist-test-slave-5l98 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:54.864172 1292 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:08:54.865686 1292 webserver.cc:460] Webserver started at http://127.0.97.3:41749/ using document root <none> and password file <none>
I20260924 03:08:54.866400 1292 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:54.866544 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:54.866851 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:54.870036 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "36f8ee07fa2640ec9c9a8c928dbb5ce0"
format_stamp: "Formatted at 2026-09-24 03:08:54 on dist-test-slave-5l98"
server_key: "a87781443c72390177de8bd287964331"
server_key_iv: "6d90c6f28bec309cf2f7f8fae38d66cb"
server_key_version: "encryptionkey@0"
I20260924 03:08:54.870829 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "36f8ee07fa2640ec9c9a8c928dbb5ce0"
format_stamp: "Formatted at 2026-09-24 03:08:54 on dist-test-slave-5l98"
server_key: "a87781443c72390177de8bd287964331"
server_key_iv: "6d90c6f28bec309cf2f7f8fae38d66cb"
server_key_version: "encryptionkey@0"
I20260924 03:08:54.876638 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 03:08:54.880436 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:54.882018 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 03:08:54.882196 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "36f8ee07fa2640ec9c9a8c928dbb5ce0"
format_stamp: "Formatted at 2026-09-24 03:08:54 on dist-test-slave-5l98"
server_key: "a87781443c72390177de8bd287964331"
server_key_iv: "6d90c6f28bec309cf2f7f8fae38d66cb"
server_key_version: "encryptionkey@0"
I20260924 03:08:54.882448 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:54.896835 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:54.901432 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:54.901738 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:54.902717 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:54.904109 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:54.904214 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:54.904362 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:54.904448 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:54.939994 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42873
I20260924 03:08:54.940033 1419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42873 every 8 connection(s)
I20260924 03:08:54.941282 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:08:54.951910 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 1292
I20260924 03:08:54.952154 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:08:54.952759 388 external_mini_cluster.cc:1428] Setting key 825dab6e1658132b5df4a1f8adbc691b
I20260924 03:08:54.954560 1153 heartbeater.cc:503] Master 127.0.97.62:34425 was elected leader, sending a full tablet report...
Sep 24 03:08:54 dist-test-slave-5l98 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:08:54.963383 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:54.943843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32949 (local address 127.0.97.62:34425)
0924 03:08:54.944065 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:54.944070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:54.945248 (+ 1178us) server_negotiation.cc:407] Connection header received
0924 03:08:54.947072 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:54.947078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:54.947171 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:54.947284 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:54.948926 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:54.949706 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:54.951419 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:54.952048 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:54.952180 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:54.957653 (+ 5473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:54.957659 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:54.957666 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:54.957701 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:54.960186 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:54.961048 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:54.961054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:54.961057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:54.961180 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:54.961620 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:54.961625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:54.961628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:54.961901 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:54.962056 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:54.962783 (+ 727us) server_negotiation.cc:299] Negotiation successful
0924 03:08:54.963009 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 03:08:54.964392 1422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:54.944253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34425 (local address 127.0.97.3:32949)
0924 03:08:54.945075 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:54.945113 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:54.946769 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:54.947523 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:54.947532 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:54.948068 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:54.948752 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:54.948773 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:54.949863 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:54.949873 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:54.951239 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:54.951250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:54.952540 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:54.952549 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:54.952900 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:54.954001 (+ 1101us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:54.954037 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:54.957408 (+ 3371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:54.960353 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:54.960363 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:54.960393 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:54.960905 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:54.961293 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:54.961298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:54.961302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:54.961498 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:54.962030 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:54.962039 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:54.962403 (+ 364us) client_negotiation.cc:769] Sending connection context
0924 03:08:54.962675 (+ 272us) client_negotiation.cc:240] Negotiation successful
0924 03:08:54.963264 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":331,"threads_started":1}
I20260924 03:08:54.966064 1420 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34425
I20260924 03:08:54.966337 1420 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:54.967067 1420 heartbeater.cc:511] Master 127.0.97.62:34425 requested a full tablet report, sending...
I20260924 03:08:54.968859 963 ts_manager.cc:194] Registered new tserver with Master: 36f8ee07fa2640ec9c9a8c928dbb5ce0 (127.0.97.3:42873)
I20260924 03:08:54.969791 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:32949
I20260924 03:08:54.981603 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:08:55.001473 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:54.988224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46042 (local address 127.0.97.62:34425)
0924 03:08:54.988394 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:54.988398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:54.988997 (+ 599us) server_negotiation.cc:407] Connection header received
0924 03:08:54.989136 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:54.989140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:54.989206 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:54.989294 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:54.991145 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:54.991965 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:54.993401 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:54.993797 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:54.993895 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:54.996176 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:54.996181 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:54.996184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:54.996214 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:54.998899 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:54.999624 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:54.999628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:54.999630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:54.999744 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:55.000079 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:55.000082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:55.000084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:55.000438 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:55.000721 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:55.001042 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 03:08:55.001199 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:08:55.008359 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:08:55.015825 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:08:55.052240 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.028639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40221 (local address 127.0.0.1:37010)
0924 03:08:55.029381 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:55.029422 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:55.029601 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:55.042485 (+ 12884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:55.042497 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:55.042549 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:55.043014 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:55.043027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.045239 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.045250 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:55.048262 (+ 3012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:55.048282 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.050659 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.050666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:55.050861 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.050919 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 03:08:55.051059 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 03:08:55.051295 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":65,"thread_start_us":1125,"threads_started":1}
I20260924 03:08:55.052591 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.028768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37010 (local address 127.0.97.2:40221)
0924 03:08:55.041337 (+ 12569us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:55.041348 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:55.041384 (+ 36us) server_negotiation.cc:407] Connection header received
0924 03:08:55.041527 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:55.041539 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:55.041873 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:55.042149 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:55.043233 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.045036 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.048518 (+ 3482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.050139 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.050287 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.050537 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:55.051986 (+ 1449us) server_negotiation.cc:299] Negotiation successful
0924 03:08:55.052182 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12411,"thread_start_us":12317,"threads_started":1}
I20260924 03:08:55.052749 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.029208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42873 (local address 127.0.0.1:46464)
0924 03:08:55.040164 (+ 10956us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:55.040205 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:55.040432 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:55.045154 (+ 4722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:55.045159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:55.045185 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:55.045730 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:55.045744 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.047444 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.047451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:55.049485 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:55.049500 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.051376 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.051381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:55.052049 (+ 668us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.052083 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:08:55.052168 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 03:08:55.052329 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10702,"mutex_wait_us":24,"thread_start_us":11318,"threads_started":1}
I20260924 03:08:55.053689 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.029606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46464 (local address 127.0.97.3:42873)
0924 03:08:55.044606 (+ 15000us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:55.044613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:55.044640 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:08:55.044717 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:55.044722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:55.044895 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:55.045174 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:55.045965 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.047298 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.049683 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.051152 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.052890 (+ 1738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.053080 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:55.053174 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 03:08:55.053356 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14806,"thread_start_us":14670,"threads_started":1}
I20260924 03:08:55.053844 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.031511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54970 (local address 127.0.97.1:37693)
0924 03:08:55.041837 (+ 10326us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:55.041847 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:55.042713 (+ 866us) server_negotiation.cc:407] Connection header received
0924 03:08:55.042839 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:55.042851 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:55.043156 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:55.043401 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:55.044579 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.046488 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.049232 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.051010 (+ 1778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.051151 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.051376 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:55.053341 (+ 1965us) server_negotiation.cc:299] Negotiation successful
0924 03:08:55.053520 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10154,"thread_start_us":9985,"threads_started":1}
I20260924 03:08:55.054421 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.030810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37693 (local address 127.0.0.1:54970)
0924 03:08:55.042298 (+ 11488us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:55.042319 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:55.042538 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:55.043646 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:55.043652 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:55.043678 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:55.044300 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:55.044318 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.046688 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.046695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:55.049019 (+ 2324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:55.049031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.052001 (+ 2970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.052007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:55.052204 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.052243 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 03:08:55.053254 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0924 03:08:55.053395 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11348,"mutex_wait_us":5,"thread_start_us":11202,"threads_started":1}
I20260924 03:08:55.057348 1222 tablet_service.cc:1467] Processing CreateTablet for tablet b6f98b4033d44ff2bd1c5d323e2939f8 (DEFAULT_TABLE table=test-table [id=19b1877c9bb34c8085a0132bc07edc92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:55.057524 1355 tablet_service.cc:1467] Processing CreateTablet for tablet b6f98b4033d44ff2bd1c5d323e2939f8 (DEFAULT_TABLE table=test-table [id=19b1877c9bb34c8085a0132bc07edc92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:55.058070 1088 tablet_service.cc:1467] Processing CreateTablet for tablet b6f98b4033d44ff2bd1c5d323e2939f8 (DEFAULT_TABLE table=test-table [id=19b1877c9bb34c8085a0132bc07edc92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:55.058967 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6f98b4033d44ff2bd1c5d323e2939f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:55.059036 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6f98b4033d44ff2bd1c5d323e2939f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:55.059557 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6f98b4033d44ff2bd1c5d323e2939f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:55.072366 1438 tablet_bootstrap.cc:492] T b6f98b4033d44ff2bd1c5d323e2939f8 P 1ef8dd8eedaa4c7e9ab9aec60c17cc70: Bootstrap starting.
I20260924 03:08:55.076534 1439 tablet_bootstrap.cc:492] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395: Bootstrap starting.
I20260924 03:08:55.078022 1438 tablet_bootstrap.cc:654] T b6f98b4033d44ff2bd1c5d323e2939f8 P 1ef8dd8eedaa4c7e9ab9aec60c17cc70: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:55.078109 1440 tablet_bootstrap.cc:492] T b6f98b4033d44ff2bd1c5d323e2939f8 P 36f8ee07fa2640ec9c9a8c928dbb5ce0: Bootstrap starting.
I20260924 03:08:55.079581 1438 log.cc:824] T b6f98b4033d44ff2bd1c5d323e2939f8 P 1ef8dd8eedaa4c7e9ab9aec60c17cc70: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:55.080790 1439 tablet_bootstrap.cc:654] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:55.081756 1440 tablet_bootstrap.cc:654] T b6f98b4033d44ff2bd1c5d323e2939f8 P 36f8ee07fa2640ec9c9a8c928dbb5ce0: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:55.082700 1439 log.cc:824] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:55.083128 1440 log.cc:824] T b6f98b4033d44ff2bd1c5d323e2939f8 P 36f8ee07fa2640ec9c9a8c928dbb5ce0: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:55.083648 1438 tablet_bootstrap.cc:492] T b6f98b4033d44ff2bd1c5d323e2939f8 P 1ef8dd8eedaa4c7e9ab9aec60c17cc70: No bootstrap required, opened a new log
I20260924 03:08:55.084079 1438 ts_tablet_manager.cc:1397] T b6f98b4033d44ff2bd1c5d323e2939f8 P 1ef8dd8eedaa4c7e9ab9aec60c17cc70: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260924 03:08:55.091626 1440 tablet_bootstrap.cc:492] T b6f98b4033d44ff2bd1c5d323e2939f8 P 36f8ee07fa2640ec9c9a8c928dbb5ce0: No bootstrap required, opened a new log
I20260924 03:08:55.091619 1439 tablet_bootstrap.cc:492] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395: No bootstrap required, opened a new log
I20260924 03:08:55.091930 1439 ts_tablet_manager.cc:1397] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20260924 03:08:55.091918 1440 ts_tablet_manager.cc:1397] T b6f98b4033d44ff2bd1c5d323e2939f8 P 36f8ee07fa2640ec9c9a8c928dbb5ce0: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260924 03:08:55.095410 1438 raft_consensus.cc:348] T b6f98b4033d44ff2bd1c5d323e2939f8 P 1ef8dd8eedaa4c7e9ab9aec60c17cc70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef8dd8eedaa4c7e9ab9aec60c17cc70" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40221 } } peers { permanent_uuid: "4938313f3b3d4a3eb5ace185bcad3395" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37693 } } peers { permanent_uuid: "36f8ee07fa2640ec9c9a8c928dbb5ce0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42873 } }
I20260924 03:08:55.095921 1438 raft_consensus.cc:374] T b6f98b4033d44ff2bd1c5d323e2939f8 P 1ef8dd8eedaa4c7e9ab9aec60c17cc70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:55.096033 1438 raft_consensus.cc:729] T b6f98b4033d44ff2bd1c5d323e2939f8 P 1ef8dd8eedaa4c7e9ab9aec60c17cc70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ef8dd8eedaa4c7e9ab9aec60c17cc70, State: Initialized, Role: FOLLOWER
I20260924 03:08:55.096740 1438 consensus_queue.cc:260] T b6f98b4033d44ff2bd1c5d323e2939f8 P 1ef8dd8eedaa4c7e9ab9aec60c17cc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef8dd8eedaa4c7e9ab9aec60c17cc70" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40221 } } peers { permanent_uuid: "4938313f3b3d4a3eb5ace185bcad3395" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37693 } } peers { permanent_uuid: "36f8ee07fa2640ec9c9a8c928dbb5ce0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42873 } }
I20260924 03:08:55.097903 1287 heartbeater.cc:503] Master 127.0.97.62:34425 was elected leader, sending a full tablet report...
I20260924 03:08:55.099895 1438 ts_tablet_manager.cc:1428] T b6f98b4033d44ff2bd1c5d323e2939f8 P 1ef8dd8eedaa4c7e9ab9aec60c17cc70: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260924 03:08:55.103652 1440 raft_consensus.cc:348] T b6f98b4033d44ff2bd1c5d323e2939f8 P 36f8ee07fa2640ec9c9a8c928dbb5ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef8dd8eedaa4c7e9ab9aec60c17cc70" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40221 } } peers { permanent_uuid: "4938313f3b3d4a3eb5ace185bcad3395" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37693 } } peers { permanent_uuid: "36f8ee07fa2640ec9c9a8c928dbb5ce0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42873 } }
I20260924 03:08:55.103652 1439 raft_consensus.cc:348] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef8dd8eedaa4c7e9ab9aec60c17cc70" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40221 } } peers { permanent_uuid: "4938313f3b3d4a3eb5ace185bcad3395" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37693 } } peers { permanent_uuid: "36f8ee07fa2640ec9c9a8c928dbb5ce0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42873 } }
I20260924 03:08:55.104135 1439 raft_consensus.cc:374] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:55.104135 1440 raft_consensus.cc:374] T b6f98b4033d44ff2bd1c5d323e2939f8 P 36f8ee07fa2640ec9c9a8c928dbb5ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:55.104241 1440 raft_consensus.cc:729] T b6f98b4033d44ff2bd1c5d323e2939f8 P 36f8ee07fa2640ec9c9a8c928dbb5ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f8ee07fa2640ec9c9a8c928dbb5ce0, State: Initialized, Role: FOLLOWER
I20260924 03:08:55.104244 1439 raft_consensus.cc:729] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4938313f3b3d4a3eb5ace185bcad3395, State: Initialized, Role: FOLLOWER
I20260924 03:08:55.104923 1440 consensus_queue.cc:260] T b6f98b4033d44ff2bd1c5d323e2939f8 P 36f8ee07fa2640ec9c9a8c928dbb5ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef8dd8eedaa4c7e9ab9aec60c17cc70" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40221 } } peers { permanent_uuid: "4938313f3b3d4a3eb5ace185bcad3395" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37693 } } peers { permanent_uuid: "36f8ee07fa2640ec9c9a8c928dbb5ce0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42873 } }
I20260924 03:08:55.104934 1439 consensus_queue.cc:260] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef8dd8eedaa4c7e9ab9aec60c17cc70" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40221 } } peers { permanent_uuid: "4938313f3b3d4a3eb5ace185bcad3395" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37693 } } peers { permanent_uuid: "36f8ee07fa2640ec9c9a8c928dbb5ce0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42873 } }
I20260924 03:08:55.106143 1420 heartbeater.cc:503] Master 127.0.97.62:34425 was elected leader, sending a full tablet report...
I20260924 03:08:55.106901 1440 ts_tablet_manager.cc:1428] T b6f98b4033d44ff2bd1c5d323e2939f8 P 36f8ee07fa2640ec9c9a8c928dbb5ce0: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260924 03:08:55.107108 1439 ts_tablet_manager.cc:1428] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260924 03:08:55.171371 1154 tablet.cc:2369] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:08:55.190160 1288 tablet.cc:2369] T b6f98b4033d44ff2bd1c5d323e2939f8 P 1ef8dd8eedaa4c7e9ab9aec60c17cc70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:08:55.192740 1421 tablet.cc:2369] T b6f98b4033d44ff2bd1c5d323e2939f8 P 36f8ee07fa2640ec9c9a8c928dbb5ce0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:08:55.265883 1445 raft_consensus.cc:482] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:08:55.266232 1445 raft_consensus.cc:504] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef8dd8eedaa4c7e9ab9aec60c17cc70" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40221 } } peers { permanent_uuid: "4938313f3b3d4a3eb5ace185bcad3395" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37693 } } peers { permanent_uuid: "36f8ee07fa2640ec9c9a8c928dbb5ce0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42873 } }
I20260924 03:08:55.268393 1445 leader_election.cc:288] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ef8dd8eedaa4c7e9ab9aec60c17cc70 (127.0.97.2:40221), 36f8ee07fa2640ec9c9a8c928dbb5ce0 (127.0.97.3:42873)
I20260924 03:08:55.278046 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.269015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35209 (local address 127.0.97.2:40221)
0924 03:08:55.269207 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:55.269213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:55.269982 (+ 769us) server_negotiation.cc:407] Connection header received
0924 03:08:55.270246 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:55.270252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:55.270332 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:55.270435 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:55.271303 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.272424 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.275491 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.276796 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.276962 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.277063 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:55.277617 (+ 554us) server_negotiation.cc:299] Negotiation successful
0924 03:08:55.277754 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:08:55.278039 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.269003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50225 (local address 127.0.97.3:42873)
0924 03:08:55.269207 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:55.269213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:55.270104 (+ 891us) server_negotiation.cc:407] Connection header received
0924 03:08:55.270248 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:55.270253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:55.270332 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:55.270435 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:55.271302 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.272440 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.275496 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.276859 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.276971 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.277063 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:55.277617 (+ 554us) server_negotiation.cc:299] Negotiation successful
0924 03:08:55.277824 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 03:08:55.278213 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.268992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42873 (local address 127.0.97.1:50225)
0924 03:08:55.269966 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:55.270013 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:55.270137 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:55.270495 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:55.270500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:55.270537 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:55.271082 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:55.271094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.272628 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.272634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:55.275292 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:55.275308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.277121 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.277129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:55.277383 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.277452 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 03:08:55.277555 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 03:08:55.277734 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":151,"spinlock_wait_cycles":1280,"thread_start_us":428,"threads_started":1}
I20260924 03:08:55.278453 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.268870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40221 (local address 127.0.97.1:35209)
0924 03:08:55.269836 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:55.269858 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:55.270054 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:55.270495 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:55.270500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:55.270543 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:55.271082 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:55.271094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.272629 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.272634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:55.275292 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:55.275308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.277138 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.277143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:55.277387 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.277452 (+ 65us) client_negotiation.cc:769] Sending connection context
0924 03:08:55.277555 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 03:08:55.277734 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":243,"spinlock_wait_cycles":6400,"thread_start_us":545,"threads_started":1}
I20260924 03:08:55.279593 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6f98b4033d44ff2bd1c5d323e2939f8" candidate_uuid: "4938313f3b3d4a3eb5ace185bcad3395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ef8dd8eedaa4c7e9ab9aec60c17cc70" is_pre_election: true
I20260924 03:08:55.279628 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6f98b4033d44ff2bd1c5d323e2939f8" candidate_uuid: "4938313f3b3d4a3eb5ace185bcad3395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36f8ee07fa2640ec9c9a8c928dbb5ce0" is_pre_election: true
I20260924 03:08:55.280076 1242 raft_consensus.cc:2454] T b6f98b4033d44ff2bd1c5d323e2939f8 P 1ef8dd8eedaa4c7e9ab9aec60c17cc70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4938313f3b3d4a3eb5ace185bcad3395 in term 0.
I20260924 03:08:55.280072 1375 raft_consensus.cc:2454] T b6f98b4033d44ff2bd1c5d323e2939f8 P 36f8ee07fa2640ec9c9a8c928dbb5ce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4938313f3b3d4a3eb5ace185bcad3395 in term 0.
I20260924 03:08:55.280746 1041 leader_election.cc:302] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1ef8dd8eedaa4c7e9ab9aec60c17cc70, 36f8ee07fa2640ec9c9a8c928dbb5ce0, 4938313f3b3d4a3eb5ace185bcad3395; no voters:
I20260924 03:08:55.281193 1445 raft_consensus.cc:2781] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:08:55.281339 1445 raft_consensus.cc:482] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:08:55.281440 1445 raft_consensus.cc:3037] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:55.284061 1445 raft_consensus.cc:504] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef8dd8eedaa4c7e9ab9aec60c17cc70" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40221 } } peers { permanent_uuid: "4938313f3b3d4a3eb5ace185bcad3395" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37693 } } peers { permanent_uuid: "36f8ee07fa2640ec9c9a8c928dbb5ce0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42873 } }
I20260924 03:08:55.284734 1445 leader_election.cc:288] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395 [CANDIDATE]: Term 1 election: Requested vote from peers 1ef8dd8eedaa4c7e9ab9aec60c17cc70 (127.0.97.2:40221), 36f8ee07fa2640ec9c9a8c928dbb5ce0 (127.0.97.3:42873)
I20260924 03:08:55.285223 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6f98b4033d44ff2bd1c5d323e2939f8" candidate_uuid: "4938313f3b3d4a3eb5ace185bcad3395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36f8ee07fa2640ec9c9a8c928dbb5ce0"
I20260924 03:08:55.285305 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6f98b4033d44ff2bd1c5d323e2939f8" candidate_uuid: "4938313f3b3d4a3eb5ace185bcad3395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ef8dd8eedaa4c7e9ab9aec60c17cc70"
I20260924 03:08:55.285444 1375 raft_consensus.cc:3037] T b6f98b4033d44ff2bd1c5d323e2939f8 P 36f8ee07fa2640ec9c9a8c928dbb5ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:55.285553 1242 raft_consensus.cc:3037] T b6f98b4033d44ff2bd1c5d323e2939f8 P 1ef8dd8eedaa4c7e9ab9aec60c17cc70 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:55.287798 1375 raft_consensus.cc:2454] T b6f98b4033d44ff2bd1c5d323e2939f8 P 36f8ee07fa2640ec9c9a8c928dbb5ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4938313f3b3d4a3eb5ace185bcad3395 in term 1.
I20260924 03:08:55.287968 1242 raft_consensus.cc:2454] T b6f98b4033d44ff2bd1c5d323e2939f8 P 1ef8dd8eedaa4c7e9ab9aec60c17cc70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4938313f3b3d4a3eb5ace185bcad3395 in term 1.
I20260924 03:08:55.288225 1044 leader_election.cc:302] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36f8ee07fa2640ec9c9a8c928dbb5ce0, 4938313f3b3d4a3eb5ace185bcad3395; no voters:
I20260924 03:08:55.288583 1445 raft_consensus.cc:2781] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:55.289047 1445 raft_consensus.cc:686] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395 [term 1 LEADER]: Becoming Leader. State: Replica: 4938313f3b3d4a3eb5ace185bcad3395, State: Running, Role: LEADER
I20260924 03:08:55.289507 1445 consensus_queue.cc:237] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef8dd8eedaa4c7e9ab9aec60c17cc70" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40221 } } peers { permanent_uuid: "4938313f3b3d4a3eb5ace185bcad3395" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37693 } } peers { permanent_uuid: "36f8ee07fa2640ec9c9a8c928dbb5ce0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42873 } }
I20260924 03:08:55.294526 963 catalog_manager.cc:5495] T b6f98b4033d44ff2bd1c5d323e2939f8 P 4938313f3b3d4a3eb5ace185bcad3395 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4938313f3b3d4a3eb5ace185bcad3395 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "4938313f3b3d4a3eb5ace185bcad3395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef8dd8eedaa4c7e9ab9aec60c17cc70" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4938313f3b3d4a3eb5ace185bcad3395" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36f8ee07fa2640ec9c9a8c928dbb5ce0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42873 } health_report { overall_health: UNKNOWN } } }
Sep 24 03:08:55 dist-test-slave-5l98 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:08:55.359498 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.342176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54984 (local address 127.0.97.1:37693)
0924 03:08:55.342402 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:55.342410 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:55.342940 (+ 530us) server_negotiation.cc:407] Connection header received
0924 03:08:55.343085 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:55.343089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:55.343152 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:55.343268 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:55.344754 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.345527 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.346787 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.347196 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.347515 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.352475 (+ 4960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:55.352549 (+ 74us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:55.352574 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:55.352616 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:55.356002 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:55.356960 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:55.356970 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:55.356972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:55.357127 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:55.357687 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:55.357692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:55.357694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:55.358183 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:55.358452 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:55.358929 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 03:08:55.359117 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 03:08:55 dist-test-slave-5l98 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:08:55 dist-test-slave-5l98 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:08:55.409914 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.394147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.0.97.1:37693)
0924 03:08:55.394445 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:55.394452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:55.394731 (+ 279us) server_negotiation.cc:407] Connection header received
0924 03:08:55.394891 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:55.394896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:55.394963 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:55.395061 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:55.396581 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.397457 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.398702 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.399274 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.399393 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.403959 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:55.403964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:55.403968 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:55.404004 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:55.406870 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:55.407638 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:55.407642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:55.407645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:55.407848 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:55.408293 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:55.408297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:55.408299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:55.408664 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:55.408967 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:55.409249 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 03:08:55.409427 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
Sep 24 03:08:55 dist-test-slave-5l98 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:08:55 dist-test-slave-5l98 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:08:55.459506 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.444089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54998 (local address 127.0.97.1:37693)
0924 03:08:55.444349 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:55.444357 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:55.444730 (+ 373us) server_negotiation.cc:407] Connection header received
0924 03:08:55.444947 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:55.444954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:55.445035 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:55.445147 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:55.446524 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.447254 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.448514 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.448939 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.449081 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.453482 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:55.453488 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:55.453492 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:55.453533 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:55.456742 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:55.457544 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:55.457549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:55.457551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:55.457678 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:55.458132 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:55.458135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:55.458137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:55.458505 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:55.458692 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:55.458964 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 03:08:55.459170 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 03:08:55.461447 1068 tablet_service.cc:2908] Rejecting scan request for tablet b6f98b4033d44ff2bd1c5d323e2939f8: Uninitialized: clean time has not yet been initialized
I20260924 03:08:55.464510 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 1025
I20260924 03:08:55.478832 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.465490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.0.97.1:37693)
0924 03:08:55.465708 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:55.465718 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:55.466096 (+ 378us) server_negotiation.cc:407] Connection header received
0924 03:08:55.466243 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:55.466247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:55.466323 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:55.466499 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:55.468257 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.469106 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.470364 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.470779 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.470922 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.473397 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:55.473402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:55.473405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:55.473439 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:55.476131 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:55.476872 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:55.476880 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:55.476886 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:55.477018 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:55.477384 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:55.477389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:55.477392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:55.477653 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:55.477905 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:55.478254 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 03:08:55.478474 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:08:55.479727 1148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:08:55.573589 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 1025
I20260924 03:08:55.591761 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 1159
Sep 24 03:08:55 dist-test-slave-5l98 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 03:08:55.607409 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.592736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37024 (local address 127.0.97.2:40221)
0924 03:08:55.592934 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:55.592941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:55.592964 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:08:55.593034 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:55.593038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:55.593095 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:55.593215 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:55.594415 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.595146 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.596437 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.596938 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.597079 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.601256 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:55.601268 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:55.601311 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:55.601345 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:55.604446 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:55.605189 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:55.605197 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:55.605200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:55.605361 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:55.605806 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:55.605810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:55.605812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:55.606195 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:55.606433 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:55.606771 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 03:08:55.607033 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 03:08:55.608232 1282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:08:55.698583 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 1159
I20260924 03:08:55.715437 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 1292
Sep 24 03:08:55 dist-test-slave-5l98 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 03:08:55.731429 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.716212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46470 (local address 127.0.97.3:42873)
0924 03:08:55.716416 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:55.716422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:55.716444 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:08:55.716625 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:55.716629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:55.716692 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:55.716898 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:55.718097 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.718794 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.720108 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.720550 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.720683 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.725426 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:55.725441 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:55.725456 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:55.725498 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:55.728597 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:55.729397 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:55.729405 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:55.729407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:55.729548 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:55.729978 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:55.729982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:55.729984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:55.730374 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:55.730599 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:55.730982 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 03:08:55.731158 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:08:55.732146 1415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:08:55.822642 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 1292
I20260924 03:08:55.838689 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 932
I20260924 03:08:55.840052 994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:08:55.951669 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 932
2026-09-24T03:08:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4548 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:08:56 dist-test-slave-5l98 krb5kdc[1468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:08:56 dist-test-slave-5l98 krb5kdc[1468](info): set up 2 sockets
Sep 24 03:08:56 dist-test-slave-5l98 krb5kdc[1468](info): commencing operation
krb5kdc: starting...
W20260924 03:08:57.136664 388 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:08:57 dist-test-slave-5l98 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:08:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:08:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:08:57.332647 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40325
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:57.661182 1481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:57.661477 1481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:57.661628 1481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:57.669097 1481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:57.669212 1481 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 03:08:57.669252 1481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:57.669348 1481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:57.669420 1481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:57.669512 1481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:08:57.679828 1481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40325
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:08:57.681289 1481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:57.682618 1481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:57.695044 1486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:57.694976 1487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:57.694993 1489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:57.696092 1481 server_base.cc:1034] running on GCE node
I20260924 03:08:57.697000 1481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:57.698508 1481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:57.699772 1481 hybrid_clock.cc:648] HybridClock initialized: now 1790219337699702 us; error 48 us; skew 500 ppm
I20260924 03:08:57.700191 1481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:08:57 dist-test-slave-5l98 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:57.712323 1481 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:08:57.714073 1481 webserver.cc:460] Webserver started at http://127.0.97.62:40043/ using document root <none> and password file <none>
I20260924 03:08:57.714810 1481 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:57.714903 1481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:57.715196 1481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:57.718253 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "6500a26d98d24491be09013b0b93ebc7"
format_stamp: "Formatted at 2026-09-24 03:08:57 on dist-test-slave-5l98"
server_key: "c645168ed0676dbc8901d039fef2cf8f"
server_key_iv: "e2bfae30d254a65201b3cb54e764ac32"
server_key_version: "encryptionkey@0"
I20260924 03:08:57.719017 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "6500a26d98d24491be09013b0b93ebc7"
format_stamp: "Formatted at 2026-09-24 03:08:57 on dist-test-slave-5l98"
server_key: "c645168ed0676dbc8901d039fef2cf8f"
server_key_iv: "e2bfae30d254a65201b3cb54e764ac32"
server_key_version: "encryptionkey@0"
I20260924 03:08:57.724246 1481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:08:57.727880 1496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:57.729486 1481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:08:57.729656 1481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "6500a26d98d24491be09013b0b93ebc7"
format_stamp: "Formatted at 2026-09-24 03:08:57 on dist-test-slave-5l98"
server_key: "c645168ed0676dbc8901d039fef2cf8f"
server_key_iv: "e2bfae30d254a65201b3cb54e764ac32"
server_key_version: "encryptionkey@0"
I20260924 03:08:57.729892 1481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:57.766412 1481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:57.771296 1481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:57.771595 1481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:57.794157 1481 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40325
I20260924 03:08:57.794176 1557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40325 every 8 connection(s)
I20260924 03:08:57.795617 1481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:08:57.799612 1558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:57.801848 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 1481
I20260924 03:08:57.802065 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:08:57.802829 388 external_mini_cluster.cc:1428] Setting key ec6f3ca4fa4d4796a32bfa13d4d8e5a5
I20260924 03:08:57.812872 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7: Bootstrap starting.
I20260924 03:08:57.817091 1558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:57.818509 1558 log.cc:824] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:08:57 dist-test-slave-5l98 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:08:57.822221 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7: No bootstrap required, opened a new log
I20260924 03:08:57.828660 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:57.805109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39544 (local address 127.0.97.62:40325)
0924 03:08:57.805991 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:57.806003 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:57.806049 (+ 46us) server_negotiation.cc:407] Connection header received
0924 03:08:57.807801 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:57.807825 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:57.808234 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:57.808757 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:57.810489 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:57.811929 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:57.813710 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:57.814603 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:57.814780 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:57.820226 (+ 5446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:57.820235 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:57.820261 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:57.820298 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:57.824140 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:57.825057 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:57.825064 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:57.825071 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:57.825259 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:57.825681 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:57.825685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:57.825687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:57.826240 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:57.826586 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:57.827021 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 03:08:57.827471 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":442,"threads_started":1}
I20260924 03:08:57.830683 1558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6500a26d98d24491be09013b0b93ebc7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40325 } }
I20260924 03:08:57.831068 1558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:57.831195 1558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6500a26d98d24491be09013b0b93ebc7, State: Initialized, Role: FOLLOWER
I20260924 03:08:57.831854 1558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6500a26d98d24491be09013b0b93ebc7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40325 } }
I20260924 03:08:57.832106 1558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:08:57.832253 1558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:08:57.832432 1558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:57.835352 1558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6500a26d98d24491be09013b0b93ebc7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40325 } }
I20260924 03:08:57.835902 1558 leader_election.cc:302] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6500a26d98d24491be09013b0b93ebc7; no voters:
I20260924 03:08:57.836603 1558 leader_election.cc:288] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:08:57.836752 1563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:57.837972 1563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7 [term 1 LEADER]: Becoming Leader. State: Replica: 6500a26d98d24491be09013b0b93ebc7, State: Running, Role: LEADER
I20260924 03:08:57.838565 1563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6500a26d98d24491be09013b0b93ebc7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40325 } }
I20260924 03:08:57.838912 1558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:57.841881 1565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6500a26d98d24491be09013b0b93ebc7. Latest consensus state: current_term: 1 leader_uuid: "6500a26d98d24491be09013b0b93ebc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6500a26d98d24491be09013b0b93ebc7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40325 } } }
I20260924 03:08:57.841857 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6500a26d98d24491be09013b0b93ebc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6500a26d98d24491be09013b0b93ebc7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40325 } } }
I20260924 03:08:57.842294 1565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7 [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:57.842311 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7 [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:57.849208 1572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:08:57.855835 1572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:08:57.864647 1572 catalog_manager.cc:1347] Generated new cluster ID: 75b6d96024e5406aa5fc2897f440d8c3
I20260924 03:08:57.864794 1572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:08:57.905690 1572 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:08:57.907424 1572 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:08:57.917757 1572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7: Generated new TSK 0
I20260924 03:08:57.918946 1572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 03:08:57.981585 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40325
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:58.323792 1584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:58.324151 1584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:58.324343 1584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:58.332015 1584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:58.332185 1584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:58.332329 1584 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:08:58.332461 1584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:08:58.342906 1584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:08:58.344432 1584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:58.346078 1584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:58.361490 1590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:58.361538 1589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:58.361511 1592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:58.362507 1584 server_base.cc:1034] running on GCE node
I20260924 03:08:58.363152 1584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:58.364642 1584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:58.365921 1584 hybrid_clock.cc:648] HybridClock initialized: now 1790219338365813 us; error 69 us; skew 500 ppm
I20260924 03:08:58.366591 1584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:08:58 dist-test-slave-5l98 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:58.371101 1584 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:08:58.372750 1584 webserver.cc:460] Webserver started at http://127.0.97.1:46311/ using document root <none> and password file <none>
I20260924 03:08:58.373579 1584 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:58.373665 1584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:58.374033 1584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:58.377063 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "848dacfde9424e0182b2207c8fd976af"
format_stamp: "Formatted at 2026-09-24 03:08:58 on dist-test-slave-5l98"
server_key: "9fdb70c306de6ab65fd3ca9dd77cbbda"
server_key_iv: "d645373334f7bddd889aa37300a42777"
server_key_version: "encryptionkey@0"
I20260924 03:08:58.377875 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "848dacfde9424e0182b2207c8fd976af"
format_stamp: "Formatted at 2026-09-24 03:08:58 on dist-test-slave-5l98"
server_key: "9fdb70c306de6ab65fd3ca9dd77cbbda"
server_key_iv: "d645373334f7bddd889aa37300a42777"
server_key_version: "encryptionkey@0"
I20260924 03:08:58.383383 1584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 03:08:58.387575 1599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:58.389457 1584 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:08:58.389664 1584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "848dacfde9424e0182b2207c8fd976af"
format_stamp: "Formatted at 2026-09-24 03:08:58 on dist-test-slave-5l98"
server_key: "9fdb70c306de6ab65fd3ca9dd77cbbda"
server_key_iv: "d645373334f7bddd889aa37300a42777"
server_key_version: "encryptionkey@0"
I20260924 03:08:58.389981 1584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:58.405416 1584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:58.410022 1584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:58.410332 1584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:58.413113 1584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:58.413225 1584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:58.413452 1584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:58.413532 1584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:08:58 dist-test-slave-5l98 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:08:58.439741 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.414196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34947 (local address 127.0.97.62:40325)
0924 03:08:58.414703 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:58.414707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:58.416013 (+ 1306us) server_negotiation.cc:407] Connection header received
0924 03:08:58.418583 (+ 2570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:58.418589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:58.418688 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:58.418812 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:58.421499 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.422825 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.425533 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.425938 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.426080 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:58.431051 (+ 4971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:58.431061 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:58.431068 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:58.431122 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:58.435467 (+ 4345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:58.436315 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:58.436324 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:58.436330 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:58.436554 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:58.437342 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:58.437350 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:58.437355 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:58.437794 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:58.437969 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:58.438874 (+ 905us) server_negotiation.cc:299] Negotiation successful
0924 03:08:58.439179 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":308,"threads_started":1}
I20260924 03:08:58.440658 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.414811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40325 (local address 127.0.97.1:34947)
0924 03:08:58.415710 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.415757 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:58.418220 (+ 2463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:58.419103 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:58.419114 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:58.420105 (+ 991us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:58.421170 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:58.421195 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.423025 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.423035 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:58.425353 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:58.425365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.426192 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.426200 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:58.426427 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:58.427257 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:58.427302 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:58.430706 (+ 3404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:58.435667 (+ 4961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:58.435677 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:58.435701 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:58.436163 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:58.436732 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:58.436743 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:58.436751 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:58.437136 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:58.437959 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:58.437969 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:58.438395 (+ 426us) client_negotiation.cc:769] Sending connection context
0924 03:08:58.438690 (+ 295us) client_negotiation.cc:240] Negotiation successful
0924 03:08:58.439166 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":431,"threads_started":1}
I20260924 03:08:58.462077 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.447498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45129 (local address 127.0.97.62:40325)
0924 03:08:58.447685 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:58.447689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:58.448422 (+ 733us) server_negotiation.cc:407] Connection header received
0924 03:08:58.448722 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:58.448727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:58.448819 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:58.448942 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:58.450551 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.451284 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.452705 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.453133 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.453279 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:58.455023 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:58.455037 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:58.455041 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:58.455071 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:58.458822 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:58.459788 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:58.459795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:58.459802 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:58.459985 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:58.460427 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:58.460431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:58.460433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:58.460879 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:58.461041 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:58.461470 (+ 429us) server_negotiation.cc:299] Negotiation successful
0924 03:08:58.461700 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 03:08:58.462311 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.447597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40325 (local address 127.0.97.1:45129)
0924 03:08:58.448308 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.448326 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:58.448542 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:58.449012 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:58.449019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:58.449769 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:58.450391 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:58.450403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.451466 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.451472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:58.452555 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:58.452575 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.453320 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.453326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:58.453494 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:58.454138 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:58.454155 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:58.454828 (+ 673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:58.459025 (+ 4197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:58.459037 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:58.459042 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:58.459626 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:58.460121 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:58.460124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:58.460127 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:58.460254 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:58.461002 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:58.461007 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:58.461195 (+ 188us) client_negotiation.cc:769] Sending connection context
0924 03:08:58.461353 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 03:08:58.461749 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":442,"threads_started":1}
I20260924 03:08:58.472558 1584 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36969
I20260924 03:08:58.472585 1719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36969 every 8 connection(s)
I20260924 03:08:58.473803 1584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:08:58.476056 1720 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40325
I20260924 03:08:58.476305 1720 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:58.477028 1720 heartbeater.cc:511] Master 127.0.97.62:40325 requested a full tablet report, sending...
I20260924 03:08:58.478966 1513 ts_manager.cc:194] Registered new tserver with Master: 848dacfde9424e0182b2207c8fd976af (127.0.97.1:36969)
I20260924 03:08:58.480589 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34947
I20260924 03:08:58.481889 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 1584
I20260924 03:08:58.482100 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:08:58.482899 388 external_mini_cluster.cc:1428] Setting key b5f15ae92cf4409c75f9e0b7fd5691f0
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 03:08:58.528613 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40325
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:58.857560 1724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:58.857908 1724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:58.858078 1724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:58.865581 1724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:58.865734 1724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:58.865890 1724 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:08:58.866031 1724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:08:58.876279 1724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:08:58.877686 1724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:58.879138 1724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:58.893110 1729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:58.893069 1730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:58.893149 1732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:58.895108 1724 server_base.cc:1034] running on GCE node
I20260924 03:08:58.895872 1724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:58.897082 1724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:58.898337 1724 hybrid_clock.cc:648] HybridClock initialized: now 1790219338898271 us; error 31 us; skew 500 ppm
I20260924 03:08:58.898800 1724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:08:58 dist-test-slave-5l98 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:58.902895 1724 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:08:58.904633 1724 webserver.cc:460] Webserver started at http://127.0.97.2:34559/ using document root <none> and password file <none>
I20260924 03:08:58.905403 1724 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:58.905515 1724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:58.905872 1724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:58.908936 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8"
format_stamp: "Formatted at 2026-09-24 03:08:58 on dist-test-slave-5l98"
server_key: "1198fa8f437ef78c9be0324637fab053"
server_key_iv: "1de2c15a80459767a1581b33cfb786ae"
server_key_version: "encryptionkey@0"
I20260924 03:08:58.909782 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8"
format_stamp: "Formatted at 2026-09-24 03:08:58 on dist-test-slave-5l98"
server_key: "1198fa8f437ef78c9be0324637fab053"
server_key_iv: "1de2c15a80459767a1581b33cfb786ae"
server_key_version: "encryptionkey@0"
I20260924 03:08:58.915257 1724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 03:08:58.919145 1739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:58.920933 1724 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260924 03:08:58.921121 1724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8"
format_stamp: "Formatted at 2026-09-24 03:08:58 on dist-test-slave-5l98"
server_key: "1198fa8f437ef78c9be0324637fab053"
server_key_iv: "1de2c15a80459767a1581b33cfb786ae"
server_key_version: "encryptionkey@0"
I20260924 03:08:58.921411 1724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:58.959107 1724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:58.964319 1724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:58.964670 1724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:58.967767 1724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:58.967989 1724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:58.968205 1724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:58.968318 1724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:08:58 dist-test-slave-5l98 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:08:58.993911 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.968907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37959 (local address 127.0.97.62:40325)
0924 03:08:58.971996 (+ 3089us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:58.972004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:58.972029 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:08:58.974032 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:58.974042 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:58.974144 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:58.974302 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:58.976958 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.977802 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.979679 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.980172 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.980344 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:58.986037 (+ 5693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:58.986043 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:58.986046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:58.986109 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:58.989509 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:58.990912 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:58.990918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:58.990922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:58.991061 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:58.991646 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:58.991652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:58.991655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:58.992038 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:58.992188 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:58.993256 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0924 03:08:58.993482 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2927,"thread_start_us":2809,"threads_started":1}
I20260924 03:08:58.995699 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.969447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40325 (local address 127.0.97.2:37959)
0924 03:08:58.970933 (+ 1486us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.971043 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:58.973688 (+ 2645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:58.974578 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:58.974587 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:58.975633 (+ 1046us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:58.976748 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:58.976775 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.977976 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.977982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:58.979509 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:58.979523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.980450 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.980457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:58.980873 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:58.982422 (+ 1549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:58.982473 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:58.985776 (+ 3303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:58.989733 (+ 3957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:58.989749 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:58.989778 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:58.990734 (+ 956us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:58.991191 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:58.991196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:58.991200 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:58.991526 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:58.992264 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:58.992271 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:58.992798 (+ 527us) client_negotiation.cc:769] Sending connection context
0924 03:08:58.993160 (+ 362us) client_negotiation.cc:240] Negotiation successful
0924 03:08:58.993751 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1014,"thread_start_us":664,"threads_started":1}
I20260924 03:08:59.018851 1814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:59.002097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40325 (local address 127.0.97.2:33815)
0924 03:08:59.002933 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:59.002955 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:59.003124 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:59.003584 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:59.003587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:59.004051 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:59.004344 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:59.004353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.005980 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.005988 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:59.007505 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:59.007519 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.009072 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.009079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:59.009351 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:59.010272 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:59.010362 (+ 90us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:59.011582 (+ 1220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:59.015407 (+ 3825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:59.015414 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:59.015424 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:59.016245 (+ 821us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:59.016799 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:59.016804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:59.016808 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:59.017003 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:59.017943 (+ 940us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:59.017946 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:59.018130 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 03:08:59.018249 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 03:08:59.018511 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":553,"threads_started":1}
I20260924 03:08:59.019208 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:59.002096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33815 (local address 127.0.97.62:40325)
0924 03:08:59.002439 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:59.002464 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:59.003061 (+ 597us) server_negotiation.cc:407] Connection header received
0924 03:08:59.003258 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:59.003266 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:59.003377 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:59.003533 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:59.004514 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.005793 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.007661 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.008593 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.008776 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:59.011859 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:59.011864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:59.011867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:59.011901 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:59.015211 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:59.016478 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:59.016506 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:59.016510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:59.016662 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:59.017138 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:59.017144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:59.017149 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:59.017722 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:59.017899 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:59.018404 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 03:08:59.018712 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:08:59.026304 1724 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36289
I20260924 03:08:59.026388 1859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36289 every 8 connection(s)
I20260924 03:08:59.027604 1724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:08:59.030050 1860 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40325
I20260924 03:08:59.030309 1860 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:59.031102 1860 heartbeater.cc:511] Master 127.0.97.62:40325 requested a full tablet report, sending...
I20260924 03:08:59.032585 1513 ts_manager.cc:194] Registered new tserver with Master: 9ca7f0cf6c184a4e914dd40ec2aaf2d8 (127.0.97.2:36289)
I20260924 03:08:59.033581 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37959
I20260924 03:08:59.037863 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 1724
I20260924 03:08:59.038084 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:08:59.038532 388 external_mini_cluster.cc:1428] Setting key 3bb2d0a56954dda6b1ca186c1dd09a79
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 03:08:59.081609 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40325
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:59.402084 1864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:59.402438 1864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:59.402619 1864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:59.410162 1864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:59.410329 1864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:59.410495 1864 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:08:59.410636 1864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:08:59.420475 1864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:08:59.421932 1864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:59.423323 1864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:59.437376 1872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:59.437436 1870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:59.437621 1869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:59.438153 1864 server_base.cc:1034] running on GCE node
I20260924 03:08:59.439028 1864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:59.440171 1864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:59.441442 1864 hybrid_clock.cc:648] HybridClock initialized: now 1790219339441354 us; error 62 us; skew 500 ppm
I20260924 03:08:59.441782 1864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:08:59 dist-test-slave-5l98 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:59.445654 1864 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:08:59.447408 1864 webserver.cc:460] Webserver started at http://127.0.97.3:42615/ using document root <none> and password file <none>
I20260924 03:08:59.448241 1864 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:59.448387 1864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:59.448797 1864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:59.451938 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "21f63a2a38674f61941731b363c0d380"
format_stamp: "Formatted at 2026-09-24 03:08:59 on dist-test-slave-5l98"
server_key: "0ebb8e11fecfd07701df2ac8aa8d4c0a"
server_key_iv: "c035d7c699e6dadd8f5252d3f98110ee"
server_key_version: "encryptionkey@0"
I20260924 03:08:59.452911 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "21f63a2a38674f61941731b363c0d380"
format_stamp: "Formatted at 2026-09-24 03:08:59 on dist-test-slave-5l98"
server_key: "0ebb8e11fecfd07701df2ac8aa8d4c0a"
server_key_iv: "c035d7c699e6dadd8f5252d3f98110ee"
server_key_version: "encryptionkey@0"
I20260924 03:08:59.458581 1864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:08:59.462404 1879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:59.463964 1864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:08:59.464165 1864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "21f63a2a38674f61941731b363c0d380"
format_stamp: "Formatted at 2026-09-24 03:08:59 on dist-test-slave-5l98"
server_key: "0ebb8e11fecfd07701df2ac8aa8d4c0a"
server_key_iv: "c035d7c699e6dadd8f5252d3f98110ee"
server_key_version: "encryptionkey@0"
I20260924 03:08:59.464421 1864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:59.478971 1864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:59.483556 1864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:59.483857 1864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:59.484256 1720 heartbeater.cc:503] Master 127.0.97.62:40325 was elected leader, sending a full tablet report...
I20260924 03:08:59.487124 1864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:59.487227 1864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:59.487380 1864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:59.487516 1864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 24 03:08:59 dist-test-slave-5l98 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:08:59.512439 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:59.487547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50383 (local address 127.0.97.62:40325)
0924 03:08:59.487746 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:59.487753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:59.489389 (+ 1636us) server_negotiation.cc:407] Connection header received
0924 03:08:59.492002 (+ 2613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:59.492007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:59.492096 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:59.492236 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:59.494846 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.495918 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.498736 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.499212 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.499354 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:59.503993 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:59.503999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:59.504003 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:59.504036 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:59.507873 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:59.509144 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:59.509152 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:59.509157 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:59.509379 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:59.510022 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:59.510031 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:59.510038 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:59.510550 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:59.510738 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:59.511533 (+ 795us) server_negotiation.cc:299] Negotiation successful
0924 03:08:59.511871 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 03:08:59.513171 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:59.488282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40325 (local address 127.0.97.3:50383)
0924 03:08:59.489215 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:59.489265 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:59.491319 (+ 2054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:59.492852 (+ 1533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:59.492905 (+ 53us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:59.493704 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:59.494642 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:59.494687 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.496085 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.496091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:59.498522 (+ 2431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:59.498536 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.499443 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.499450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:59.499668 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:59.500560 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:59.500595 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:59.503719 (+ 3124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:59.508095 (+ 4376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:59.508111 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:59.508131 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:59.508991 (+ 860us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:59.509546 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:59.509551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:59.509554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:59.509859 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:59.510715 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:59.510722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:59.511070 (+ 348us) client_negotiation.cc:769] Sending connection context
0924 03:08:59.511330 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 03:08:59.511751 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":485,"threads_started":1}
I20260924 03:08:59.536350 1947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:59.519388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40325 (local address 127.0.97.3:60389)
0924 03:08:59.520151 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:59.520175 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:59.520376 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:59.520799 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:59.520804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:59.521539 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:59.522155 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:59.522179 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.523396 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.523402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:59.525829 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:59.525853 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.526773 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.526783 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:59.527036 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:59.527788 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:59.527819 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:59.529627 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:59.533131 (+ 3504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:59.533139 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:59.533145 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:59.533948 (+ 803us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:59.534408 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:59.534416 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:59.534422 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:59.534616 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:59.535432 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:59.535436 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:59.535600 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 03:08:59.535727 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 03:08:59.535987 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":383,"threads_started":1}
I20260924 03:08:59.536698 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:59.519378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60389 (local address 127.0.97.62:40325)
0924 03:08:59.519565 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:59.519574 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:59.520303 (+ 729us) server_negotiation.cc:407] Connection header received
0924 03:08:59.520558 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:59.520565 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:59.520656 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:59.520777 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:59.522313 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.523217 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.526028 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.526551 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.526663 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:59.529920 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:59.529927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:59.529930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:59.529966 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:59.532933 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:59.534100 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:59.534106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:59.534111 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:59.534280 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:59.534786 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:59.534795 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:59.534801 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:59.535277 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:59.535465 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:59.535866 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 03:08:59.536156 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 03:08:59.550051 1864 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38363
I20260924 03:08:59.550081 1998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38363 every 8 connection(s)
I20260924 03:08:59.551282 1864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:08:59.553952 1999 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40325
I20260924 03:08:59.554211 1999 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:59.554885 1999 heartbeater.cc:511] Master 127.0.97.62:40325 requested a full tablet report, sending...
I20260924 03:08:59.556339 1513 ts_manager.cc:194] Registered new tserver with Master: 21f63a2a38674f61941731b363c0d380 (127.0.97.3:38363)
I20260924 03:08:59.557560 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50383
I20260924 03:08:59.559971 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 1864
I20260924 03:08:59.560191 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:08:59.560645 388 external_mini_cluster.cc:1428] Setting key 2491a43bd4e5fa5d2bf500e280a76620
I20260924 03:08:59.563977 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:08:59 dist-test-slave-5l98 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219339, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:08:59 dist-test-slave-5l98 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219339, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:08:59.613026 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:59.597250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39546 (local address 127.0.97.62:40325)
0924 03:08:59.597492 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:59.597496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:59.597961 (+ 465us) server_negotiation.cc:407] Connection header received
0924 03:08:59.598194 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:59.598199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:59.598306 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:59.598415 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:59.600166 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.601052 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.602426 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.602829 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.602938 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:59.607223 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:59.607228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:59.607232 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:59.607270 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:59.610168 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:59.610940 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:59.610945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:59.610948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:59.611080 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:59.611549 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:59.611553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:59.611556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:59.611897 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:59.612131 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:59.612589 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 03:08:59.612770 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260924 03:08:59.618541 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:08:59.626557 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:08:59.661163 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:59.641196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38363 (local address 127.0.0.1:46802)
0924 03:08:59.642624 (+ 1428us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:59.642659 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:59.642843 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:59.650560 (+ 7717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:59.650572 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:59.650619 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:59.651159 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:59.651174 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.654453 (+ 3279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.654460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:59.657461 (+ 3001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:59.657483 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.660083 (+ 2600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.660089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:59.660320 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:59.660380 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 03:08:59.660588 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 03:08:59.660791 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1210,"thread_start_us":1035,"threads_started":1}
I20260924 03:08:59.661216 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:59.641234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46802 (local address 127.0.97.3:38363)
0924 03:08:59.642042 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:59.642055 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:59.643039 (+ 984us) server_negotiation.cc:407] Connection header received
0924 03:08:59.646317 (+ 3278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:59.646332 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:59.646690 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:59.646968 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:59.651444 (+ 4476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.653488 (+ 2044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.657668 (+ 4180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.659325 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.659444 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:59.659673 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:59.660704 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0924 03:08:59.660886 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":513,"threads_started":1}
I20260924 03:08:59.663280 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:59.641773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36969 (local address 127.0.0.1:59740)
0924 03:08:59.652982 (+ 11209us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:59.653006 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:59.653152 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:59.654178 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:59.654183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:59.654209 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:59.654738 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:59.654749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.656914 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.656922 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:59.660420 (+ 3498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:59.660438 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.662322 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.662329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:59.662537 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:59.662571 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:08:59.662686 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 03:08:59.662820 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11058,"mutex_wait_us":4,"thread_start_us":11468,"threads_started":1}
I20260924 03:08:59.664155 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:59.642330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59740 (local address 127.0.97.1:36969)
0924 03:08:59.652984 (+ 10654us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:59.652995 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:59.653521 (+ 526us) server_negotiation.cc:407] Connection header received
0924 03:08:59.653645 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:59.653655 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:59.653958 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:59.654179 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:59.654991 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.656702 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.660611 (+ 3909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.662098 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.662249 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:59.662497 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:59.663639 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0924 03:08:59.663837 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10482,"thread_start_us":11330,"threads_started":1}
I20260924 03:08:59.666388 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:59.643052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36289 (local address 127.0.0.1:60758)
0924 03:08:59.658535 (+ 15483us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:59.658557 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:59.658685 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:59.659350 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:59.659355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:59.659383 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:59.659805 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:59.659816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.661729 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.661733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:59.663658 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:59.663671 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.665572 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.665578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:59.665779 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:59.665810 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 03:08:59.665910 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 03:08:59.666038 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15335,"mutex_wait_us":26,"thread_start_us":15180,"threads_started":1}
I20260924 03:08:59.666432 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:59.643320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60758 (local address 127.0.97.2:36289)
0924 03:08:59.658040 (+ 14720us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:59.658050 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:59.658677 (+ 627us) server_negotiation.cc:407] Connection header received
0924 03:08:59.658835 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:59.658844 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:59.659123 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:59.659343 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:59.660110 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.661555 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.663820 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.665345 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.665495 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:59.665747 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:59.665966 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 03:08:59.666122 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14519,"thread_start_us":16336,"threads_started":1}
I20260924 03:08:59.666146 1929 tablet_service.cc:1467] Processing CreateTablet for tablet d8209c7ee9e14fa28f4b396001c044d0 (DEFAULT_TABLE table=test-table [id=b97f0a2bed0a496793e725ee649fe354]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:59.667666 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8209c7ee9e14fa28f4b396001c044d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:59.668283 1650 tablet_service.cc:1467] Processing CreateTablet for tablet d8209c7ee9e14fa28f4b396001c044d0 (DEFAULT_TABLE table=test-table [id=b97f0a2bed0a496793e725ee649fe354]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:59.669772 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8209c7ee9e14fa28f4b396001c044d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:59.670351 1790 tablet_service.cc:1467] Processing CreateTablet for tablet d8209c7ee9e14fa28f4b396001c044d0 (DEFAULT_TABLE table=test-table [id=b97f0a2bed0a496793e725ee649fe354]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:59.671694 1790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8209c7ee9e14fa28f4b396001c044d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:59.682514 2018 tablet_bootstrap.cc:492] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af: Bootstrap starting.
I20260924 03:08:59.684602 2019 tablet_bootstrap.cc:492] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380: Bootstrap starting.
I20260924 03:08:59.686477 2018 tablet_bootstrap.cc:654] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:59.686502 2020 tablet_bootstrap.cc:492] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Bootstrap starting.
I20260924 03:08:59.688186 2019 tablet_bootstrap.cc:654] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:59.688282 2018 log.cc:824] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:59.689401 2019 log.cc:824] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:59.689675 2020 tablet_bootstrap.cc:654] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:59.690879 2020 log.cc:824] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:59.692014 2018 tablet_bootstrap.cc:492] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af: No bootstrap required, opened a new log
I20260924 03:08:59.692361 2018 ts_tablet_manager.cc:1397] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260924 03:08:59.692448 2019 tablet_bootstrap.cc:492] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380: No bootstrap required, opened a new log
I20260924 03:08:59.692724 2019 ts_tablet_manager.cc:1397] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260924 03:08:59.695333 2020 tablet_bootstrap.cc:492] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: No bootstrap required, opened a new log
I20260924 03:08:59.695657 2020 ts_tablet_manager.cc:1397] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 03:08:59.704097 2020 raft_consensus.cc:348] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } } peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } }
I20260924 03:08:59.704586 2020 raft_consensus.cc:374] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:59.704687 2020 raft_consensus.cc:729] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca7f0cf6c184a4e914dd40ec2aaf2d8, State: Initialized, Role: FOLLOWER
I20260924 03:08:59.705295 2020 consensus_queue.cc:260] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } } peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } }
I20260924 03:08:59.705513 2019 raft_consensus.cc:348] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } } peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } }
I20260924 03:08:59.705986 2019 raft_consensus.cc:374] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:59.705763 2018 raft_consensus.cc:348] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } } peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } }
I20260924 03:08:59.706089 2019 raft_consensus.cc:729] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f63a2a38674f61941731b363c0d380, State: Initialized, Role: FOLLOWER
I20260924 03:08:59.706200 2018 raft_consensus.cc:374] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:59.706284 2018 raft_consensus.cc:729] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 848dacfde9424e0182b2207c8fd976af, State: Initialized, Role: FOLLOWER
I20260924 03:08:59.706822 2019 consensus_queue.cc:260] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } } peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } }
I20260924 03:08:59.706881 2018 consensus_queue.cc:260] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } } peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } }
I20260924 03:08:59.709823 2018 ts_tablet_manager.cc:1428] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260924 03:08:59.711016 1999 heartbeater.cc:503] Master 127.0.97.62:40325 was elected leader, sending a full tablet report...
I20260924 03:08:59.711498 1860 heartbeater.cc:503] Master 127.0.97.62:40325 was elected leader, sending a full tablet report...
I20260924 03:08:59.712316 2020 ts_tablet_manager.cc:1428] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260924 03:08:59.713428 2019 ts_tablet_manager.cc:1428] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380: Time spent starting tablet: real 0.020s user 0.017s sys 0.001s
W20260924 03:08:59.726266 1721 tablet.cc:2369] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:08:59.779819 1861 tablet.cc:2369] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:08:59.802917 2000 tablet.cc:2369] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:09:00.006403 2026 raft_consensus.cc:482] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:00.006754 2026 raft_consensus.cc:504] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } } peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } }
I20260924 03:09:00.009184 2026 leader_election.cc:288] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ca7f0cf6c184a4e914dd40ec2aaf2d8 (127.0.97.2:36289), 848dacfde9424e0182b2207c8fd976af (127.0.97.1:36969)
I20260924 03:09:00.017619 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.009410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36969 (local address 127.0.97.3:55801)
0924 03:09:00.009662 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:00.009687 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:00.009823 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:00.010644 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:00.010649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:00.010691 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:00.010923 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.010939 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.012216 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.012221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.015116 (+ 2895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.015150 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.016995 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.017000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.017149 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.017176 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 03:09:00.017266 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 03:09:00.017376 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260924 03:09:00.017720 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.009833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55801 (local address 127.0.97.1:36969)
0924 03:09:00.010172 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.010176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.010230 (+ 54us) server_negotiation.cc:407] Connection header received
0924 03:09:00.010308 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.010314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.010466 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.010564 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:00.011066 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.012046 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.015551 (+ 3505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.016639 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.016833 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.016961 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.017328 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.017468 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 03:09:00.019361 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.009503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53307 (local address 127.0.97.2:36289)
0924 03:09:00.009708 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.009713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.010773 (+ 1060us) server_negotiation.cc:407] Connection header received
0924 03:09:00.010920 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.010924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.010982 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.011103 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:00.011672 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.012966 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.016774 (+ 3808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.017903 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.018059 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.018178 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.019041 (+ 863us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.019157 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 03:09:00.019611 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.009993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36289 (local address 127.0.97.3:53307)
0924 03:09:00.010651 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:00.010675 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:00.010796 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:00.011211 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:00.011214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:00.011236 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:00.011494 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.011501 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.013210 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.013218 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.016542 (+ 3324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.016564 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.018157 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.018164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.018319 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.018344 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 03:09:00.018973 (+ 629us) client_negotiation.cc:240] Negotiation successful
0924 03:09:00.019153 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":1,"thread_start_us":359,"threads_started":1}
I20260924 03:09:00.019409 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8209c7ee9e14fa28f4b396001c044d0" candidate_uuid: "21f63a2a38674f61941731b363c0d380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "848dacfde9424e0182b2207c8fd976af" is_pre_election: true
I20260924 03:09:00.020094 1675 raft_consensus.cc:2454] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21f63a2a38674f61941731b363c0d380 in term 0.
I20260924 03:09:00.020880 1880 leader_election.cc:302] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f63a2a38674f61941731b363c0d380, 848dacfde9424e0182b2207c8fd976af; no voters:
I20260924 03:09:00.020838 1815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8209c7ee9e14fa28f4b396001c044d0" candidate_uuid: "21f63a2a38674f61941731b363c0d380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" is_pre_election: true
I20260924 03:09:00.021322 2026 raft_consensus.cc:2781] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:00.021344 1815 raft_consensus.cc:2454] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21f63a2a38674f61941731b363c0d380 in term 0.
I20260924 03:09:00.021750 2026 raft_consensus.cc:482] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:00.021960 2026 raft_consensus.cc:3037] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:00.026386 2026 raft_consensus.cc:504] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } } peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } }
I20260924 03:09:00.027065 2026 leader_election.cc:288] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 [CANDIDATE]: Term 1 election: Requested vote from peers 9ca7f0cf6c184a4e914dd40ec2aaf2d8 (127.0.97.2:36289), 848dacfde9424e0182b2207c8fd976af (127.0.97.1:36969)
I20260924 03:09:00.027671 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8209c7ee9e14fa28f4b396001c044d0" candidate_uuid: "21f63a2a38674f61941731b363c0d380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "848dacfde9424e0182b2207c8fd976af"
I20260924 03:09:00.027674 1815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8209c7ee9e14fa28f4b396001c044d0" candidate_uuid: "21f63a2a38674f61941731b363c0d380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8"
I20260924 03:09:00.027904 1815 raft_consensus.cc:3037] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:00.027891 1675 raft_consensus.cc:3037] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:00.031324 1675 raft_consensus.cc:2454] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21f63a2a38674f61941731b363c0d380 in term 1.
I20260924 03:09:00.031325 1815 raft_consensus.cc:2454] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21f63a2a38674f61941731b363c0d380 in term 1.
I20260924 03:09:00.031809 1880 leader_election.cc:302] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f63a2a38674f61941731b363c0d380, 848dacfde9424e0182b2207c8fd976af; no voters:
I20260924 03:09:00.032207 2026 raft_consensus.cc:2781] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:00.032779 2026 raft_consensus.cc:686] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 [term 1 LEADER]: Becoming Leader. State: Replica: 21f63a2a38674f61941731b363c0d380, State: Running, Role: LEADER
I20260924 03:09:00.033391 2026 consensus_queue.cc:237] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } } peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } }
I20260924 03:09:00.039343 1513 catalog_manager.cc:5495] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21f63a2a38674f61941731b363c0d380 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "21f63a2a38674f61941731b363c0d380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } health_report { overall_health: UNKNOWN } } }
I20260924 03:09:00.121850 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.114285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46814 (local address 127.0.97.3:38363)
0924 03:09:00.114555 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.114562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.115355 (+ 793us) server_negotiation.cc:407] Connection header received
0924 03:09:00.115701 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.115706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.115786 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.115982 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:00.116793 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.117641 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.119361 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.119806 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.120065 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.120678 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:00.121267 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:00.121382 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.121565 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.121638 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260924 03:09:00.129972 1675 raft_consensus.cc:1261] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af [term 1 FOLLOWER]: Refusing update from remote peer 21f63a2a38674f61941731b363c0d380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:00.130631 1815 raft_consensus.cc:1261] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 1 FOLLOWER]: Refusing update from remote peer 21f63a2a38674f61941731b363c0d380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:00.131088 2028 consensus_queue.cc:1035] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:00.131637 2026 consensus_queue.cc:1035] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:00.166617 1512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:00.167047 1512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:00.171006 1512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7: Sending DeleteTablet for 3 replicas of tablet d8209c7ee9e14fa28f4b396001c044d0
I20260924 03:09:00.172382 1790 tablet_service.cc:1514] Processing DeleteTablet for tablet d8209c7ee9e14fa28f4b396001c044d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60758
I20260924 03:09:00.172405 1650 tablet_service.cc:1514] Processing DeleteTablet for tablet d8209c7ee9e14fa28f4b396001c044d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59740
I20260924 03:09:00.172626 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet d8209c7ee9e14fa28f4b396001c044d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46802
I20260924 03:09:00.173877 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:09:00.174703 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:00.186208 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 52c4ac114fe44f92b290fb68a5b03885 (DEFAULT_TABLE table=test-table [id=f2563cff2a494f24a8a357a1ace608c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:00.186553 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 52c4ac114fe44f92b290fb68a5b03885 (DEFAULT_TABLE table=test-table [id=f2563cff2a494f24a8a357a1ace608c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:00.186904 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52c4ac114fe44f92b290fb68a5b03885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:00.186735 1789 tablet_service.cc:1467] Processing CreateTablet for tablet 52c4ac114fe44f92b290fb68a5b03885 (DEFAULT_TABLE table=test-table [id=f2563cff2a494f24a8a357a1ace608c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:00.187080 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52c4ac114fe44f92b290fb68a5b03885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:00.187439 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52c4ac114fe44f92b290fb68a5b03885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:00.195007 2018 tablet_bootstrap.cc:492] T 52c4ac114fe44f92b290fb68a5b03885 P 848dacfde9424e0182b2207c8fd976af: Bootstrap starting.
I20260924 03:09:00.200843 2018 tablet_bootstrap.cc:654] T 52c4ac114fe44f92b290fb68a5b03885 P 848dacfde9424e0182b2207c8fd976af: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:00.205948 2019 tablet_bootstrap.cc:492] T 52c4ac114fe44f92b290fb68a5b03885 P 21f63a2a38674f61941731b363c0d380: Bootstrap starting.
I20260924 03:09:00.208317 2019 tablet_bootstrap.cc:654] T 52c4ac114fe44f92b290fb68a5b03885 P 21f63a2a38674f61941731b363c0d380: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:00.212770 2020 tablet_bootstrap.cc:492] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Bootstrap starting.
I20260924 03:09:00.215188 2020 tablet_bootstrap.cc:654] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:00.223129 2019 tablet_bootstrap.cc:492] T 52c4ac114fe44f92b290fb68a5b03885 P 21f63a2a38674f61941731b363c0d380: No bootstrap required, opened a new log
I20260924 03:09:00.223366 2019 ts_tablet_manager.cc:1397] T 52c4ac114fe44f92b290fb68a5b03885 P 21f63a2a38674f61941731b363c0d380: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.000s
I20260924 03:09:00.224753 2019 raft_consensus.cc:348] T 52c4ac114fe44f92b290fb68a5b03885 P 21f63a2a38674f61941731b363c0d380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } }
I20260924 03:09:00.225091 2019 raft_consensus.cc:374] T 52c4ac114fe44f92b290fb68a5b03885 P 21f63a2a38674f61941731b363c0d380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:00.225183 2019 raft_consensus.cc:729] T 52c4ac114fe44f92b290fb68a5b03885 P 21f63a2a38674f61941731b363c0d380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f63a2a38674f61941731b363c0d380, State: Initialized, Role: FOLLOWER
I20260924 03:09:00.225663 2019 consensus_queue.cc:260] T 52c4ac114fe44f92b290fb68a5b03885 P 21f63a2a38674f61941731b363c0d380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } }
I20260924 03:09:00.227620 2019 ts_tablet_manager.cc:1428] T 52c4ac114fe44f92b290fb68a5b03885 P 21f63a2a38674f61941731b363c0d380: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260924 03:09:00.241379 2018 tablet_bootstrap.cc:492] T 52c4ac114fe44f92b290fb68a5b03885 P 848dacfde9424e0182b2207c8fd976af: No bootstrap required, opened a new log
I20260924 03:09:00.244555 2020 tablet_bootstrap.cc:492] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: No bootstrap required, opened a new log
I20260924 03:09:00.244781 2020 ts_tablet_manager.cc:1397] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.006s
I20260924 03:09:00.245985 2020 raft_consensus.cc:348] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } }
I20260924 03:09:00.246320 2020 raft_consensus.cc:374] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:00.246446 2020 raft_consensus.cc:729] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca7f0cf6c184a4e914dd40ec2aaf2d8, State: Initialized, Role: FOLLOWER
I20260924 03:09:00.246842 2020 consensus_queue.cc:260] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } }
I20260924 03:09:00.252396 2020 ts_tablet_manager.cc:1428] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20260924 03:09:00.252887 2018 ts_tablet_manager.cc:1397] T 52c4ac114fe44f92b290fb68a5b03885 P 848dacfde9424e0182b2207c8fd976af: Time spent bootstrapping tablet: real 0.058s user 0.008s sys 0.004s
I20260924 03:09:00.254066 2018 raft_consensus.cc:348] T 52c4ac114fe44f92b290fb68a5b03885 P 848dacfde9424e0182b2207c8fd976af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } }
I20260924 03:09:00.254402 2018 raft_consensus.cc:374] T 52c4ac114fe44f92b290fb68a5b03885 P 848dacfde9424e0182b2207c8fd976af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:00.254554 2018 raft_consensus.cc:729] T 52c4ac114fe44f92b290fb68a5b03885 P 848dacfde9424e0182b2207c8fd976af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 848dacfde9424e0182b2207c8fd976af, State: Initialized, Role: FOLLOWER
I20260924 03:09:00.258018 2018 consensus_queue.cc:260] T 52c4ac114fe44f92b290fb68a5b03885 P 848dacfde9424e0182b2207c8fd976af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } }
I20260924 03:09:00.259023 2042 tablet_replica.cc:330] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: stopping tablet replica
I20260924 03:09:00.259634 2042 raft_consensus.cc:2229] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:00.260257 2042 raft_consensus.cc:2258] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:00.262619 2042 ts_tablet_manager.cc:1905] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:00.263324 2018 ts_tablet_manager.cc:1428] T 52c4ac114fe44f92b290fb68a5b03885 P 848dacfde9424e0182b2207c8fd976af: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20260924 03:09:00.263685 2044 tablet_replica.cc:330] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380: stopping tablet replica
I20260924 03:09:00.264151 2044 raft_consensus.cc:2229] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:09:00.264875 2044 raft_consensus.cc:2258] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:00.269001 2043 tablet_replica.cc:330] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af: stopping tablet replica
I20260924 03:09:00.269430 2043 raft_consensus.cc:2229] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:00.269841 2043 raft_consensus.cc:2258] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:00.270480 2044 ts_tablet_manager.cc:1905] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:00.271735 2042 ts_tablet_manager.cc:1918] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:00.271934 2042 log.cc:1196] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/d8209c7ee9e14fa28f4b396001c044d0
I20260924 03:09:00.272626 2042 ts_tablet_manager.cc:1939] T d8209c7ee9e14fa28f4b396001c044d0 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Deleting consensus metadata
I20260924 03:09:00.273684 2043 ts_tablet_manager.cc:1905] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:00.274750 1499 catalog_manager.cc:4841] TS 9ca7f0cf6c184a4e914dd40ec2aaf2d8 (127.0.97.2:36289): tablet d8209c7ee9e14fa28f4b396001c044d0 (table test-table [id=b97f0a2bed0a496793e725ee649fe354]) successfully deleted
I20260924 03:09:00.280174 2043 ts_tablet_manager.cc:1918] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:00.280354 2043 log.cc:1196] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/d8209c7ee9e14fa28f4b396001c044d0
I20260924 03:09:00.280444 2044 ts_tablet_manager.cc:1918] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:00.280658 2044 log.cc:1196] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/d8209c7ee9e14fa28f4b396001c044d0
I20260924 03:09:00.280951 2043 ts_tablet_manager.cc:1939] T d8209c7ee9e14fa28f4b396001c044d0 P 848dacfde9424e0182b2207c8fd976af: Deleting consensus metadata
I20260924 03:09:00.281374 2044 ts_tablet_manager.cc:1939] T d8209c7ee9e14fa28f4b396001c044d0 P 21f63a2a38674f61941731b363c0d380: Deleting consensus metadata
I20260924 03:09:00.282605 1497 catalog_manager.cc:4841] TS 848dacfde9424e0182b2207c8fd976af (127.0.97.1:36969): tablet d8209c7ee9e14fa28f4b396001c044d0 (table test-table [id=b97f0a2bed0a496793e725ee649fe354]) successfully deleted
I20260924 03:09:00.283048 1499 catalog_manager.cc:4841] TS 21f63a2a38674f61941731b363c0d380 (127.0.97.3:38363): tablet d8209c7ee9e14fa28f4b396001c044d0 (table test-table [id=b97f0a2bed0a496793e725ee649fe354]) successfully deleted
I20260924 03:09:00.294915 2024 raft_consensus.cc:482] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:00.295161 2024 raft_consensus.cc:504] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } }
I20260924 03:09:00.296805 2024 leader_election.cc:288] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 848dacfde9424e0182b2207c8fd976af (127.0.97.1:36969), 21f63a2a38674f61941731b363c0d380 (127.0.97.3:38363)
I20260924 03:09:00.306231 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.297780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33957 (local address 127.0.97.1:36969)
0924 03:09:00.297976 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.297982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.298520 (+ 538us) server_negotiation.cc:407] Connection header received
0924 03:09:00.298770 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.298776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.298860 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.298930 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:00.299633 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.300422 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.303575 (+ 3153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.305140 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.305293 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.305412 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.305762 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.305921 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 03:09:00.306294 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.297204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36969 (local address 127.0.97.2:33957)
0924 03:09:00.298406 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:00.298428 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:00.298533 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:00.299114 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:00.299117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:00.299136 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:00.299393 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.299399 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.300671 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.300675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.303384 (+ 2709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.303399 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.305386 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.305394 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.305567 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.305605 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 03:09:00.305719 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 03:09:00.305857 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"mutex_wait_us":158,"thread_start_us":858,"threads_started":1}
I20260924 03:09:00.307124 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.297100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38363 (local address 127.0.97.2:45627)
0924 03:09:00.297649 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:00.297669 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:00.297821 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:00.298424 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:00.298430 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:00.298463 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:00.298909 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.298923 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.300558 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.300565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.303384 (+ 2819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.303399 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.306232 (+ 2833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.306244 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.306476 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.306505 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 03:09:00.306595 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 03:09:00.306736 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":164,"thread_start_us":699,"threads_started":1}
I20260924 03:09:00.307125 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.297501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45627 (local address 127.0.97.3:38363)
0924 03:09:00.297706 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.297714 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.297735 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:09:00.298005 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.298014 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.298124 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.298198 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:00.299061 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.300382 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.304655 (+ 4273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.305838 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.305947 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.306051 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.306664 (+ 613us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.306822 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 03:09:00.307380 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52c4ac114fe44f92b290fb68a5b03885" candidate_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "848dacfde9424e0182b2207c8fd976af" is_pre_election: true
I20260924 03:09:00.307659 1675 raft_consensus.cc:2454] T 52c4ac114fe44f92b290fb68a5b03885 P 848dacfde9424e0182b2207c8fd976af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ca7f0cf6c184a4e914dd40ec2aaf2d8 in term 0.
I20260924 03:09:00.308167 1740 leader_election.cc:302] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 848dacfde9424e0182b2207c8fd976af, 9ca7f0cf6c184a4e914dd40ec2aaf2d8; no voters:
I20260924 03:09:00.308009 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52c4ac114fe44f92b290fb68a5b03885" candidate_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f63a2a38674f61941731b363c0d380" is_pre_election: true
I20260924 03:09:00.308583 2024 raft_consensus.cc:2781] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:00.308519 1954 raft_consensus.cc:2454] T 52c4ac114fe44f92b290fb68a5b03885 P 21f63a2a38674f61941731b363c0d380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ca7f0cf6c184a4e914dd40ec2aaf2d8 in term 0.
I20260924 03:09:00.308737 2024 raft_consensus.cc:482] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:00.308837 2024 raft_consensus.cc:3037] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:00.311395 2024 raft_consensus.cc:504] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } }
I20260924 03:09:00.312258 2024 leader_election.cc:288] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [CANDIDATE]: Term 1 election: Requested vote from peers 848dacfde9424e0182b2207c8fd976af (127.0.97.1:36969), 21f63a2a38674f61941731b363c0d380 (127.0.97.3:38363)
I20260924 03:09:00.312593 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52c4ac114fe44f92b290fb68a5b03885" candidate_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f63a2a38674f61941731b363c0d380"
I20260924 03:09:00.312618 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52c4ac114fe44f92b290fb68a5b03885" candidate_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "848dacfde9424e0182b2207c8fd976af"
I20260924 03:09:00.312800 1675 raft_consensus.cc:3037] T 52c4ac114fe44f92b290fb68a5b03885 P 848dacfde9424e0182b2207c8fd976af [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:00.312840 1954 raft_consensus.cc:3037] T 52c4ac114fe44f92b290fb68a5b03885 P 21f63a2a38674f61941731b363c0d380 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:00.316346 1675 raft_consensus.cc:2454] T 52c4ac114fe44f92b290fb68a5b03885 P 848dacfde9424e0182b2207c8fd976af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca7f0cf6c184a4e914dd40ec2aaf2d8 in term 1.
I20260924 03:09:00.316375 1954 raft_consensus.cc:2454] T 52c4ac114fe44f92b290fb68a5b03885 P 21f63a2a38674f61941731b363c0d380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca7f0cf6c184a4e914dd40ec2aaf2d8 in term 1.
I20260924 03:09:00.316826 1740 leader_election.cc:302] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 848dacfde9424e0182b2207c8fd976af, 9ca7f0cf6c184a4e914dd40ec2aaf2d8; no voters:
I20260924 03:09:00.317171 2024 raft_consensus.cc:2781] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:00.317778 2024 raft_consensus.cc:686] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 1 LEADER]: Becoming Leader. State: Replica: 9ca7f0cf6c184a4e914dd40ec2aaf2d8, State: Running, Role: LEADER
I20260924 03:09:00.318246 2024 consensus_queue.cc:237] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } }
I20260924 03:09:00.322109 1513 catalog_manager.cc:5495] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ca7f0cf6c184a4e914dd40ec2aaf2d8 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } health_report { overall_health: UNKNOWN } } }
Sep 24 03:09:00 dist-test-slave-5l98 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:00.383500 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.363386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40325 (local address 127.0.0.1:39558)
0924 03:09:00.364771 (+ 1385us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:00.364790 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:00.364941 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:00.365542 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:00.365545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:00.366807 (+ 1262us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:00.367285 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.367302 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.369399 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.369404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.370897 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.370909 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.372061 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.372065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.372325 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.373192 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:00.373212 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:00.375883 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:00.380327 (+ 4444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:00.380340 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:00.380347 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:00.381006 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:00.381471 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:00.381474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:00.381477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:00.381612 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:00.382245 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:00.382253 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:00.382412 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 03:09:00.382548 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 03:09:00.382852 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1226,"thread_start_us":918,"threads_started":1}
I20260924 03:09:00.383899 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.363389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39558 (local address 127.0.97.62:40325)
0924 03:09:00.364642 (+ 1253us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.364650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.364908 (+ 258us) server_negotiation.cc:407] Connection header received
0924 03:09:00.365099 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.365106 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.365193 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.365342 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:00.367515 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.369011 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.371070 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.371743 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.371881 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.376150 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:00.376158 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:00.376163 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:00.376204 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:00.380060 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.381142 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.381149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.381153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.381321 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.381731 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.381737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.381740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.382108 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:00.382255 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.382686 (+ 431us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.383420 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":848,"thread_start_us":362,"threads_started":1}
I20260924 03:09:00.390941 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39558:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 03:09:00.392388 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:00.400940 1790 tablet_service.cc:1467] Processing CreateTablet for tablet a7dc393276244199b04c546f78146659 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8a392f63826e440fad1e712c7a964a8c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:09:00.401211 1650 tablet_service.cc:1467] Processing CreateTablet for tablet a7dc393276244199b04c546f78146659 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8a392f63826e440fad1e712c7a964a8c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:09:00.401578 1790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7dc393276244199b04c546f78146659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:00.401809 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7dc393276244199b04c546f78146659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:00.406410 1929 tablet_service.cc:1467] Processing CreateTablet for tablet a7dc393276244199b04c546f78146659 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8a392f63826e440fad1e712c7a964a8c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:09:00.407063 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7dc393276244199b04c546f78146659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:00.410194 2018 tablet_bootstrap.cc:492] T a7dc393276244199b04c546f78146659 P 848dacfde9424e0182b2207c8fd976af: Bootstrap starting.
I20260924 03:09:00.411983 2018 tablet_bootstrap.cc:654] T a7dc393276244199b04c546f78146659 P 848dacfde9424e0182b2207c8fd976af: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:00.413714 2020 tablet_bootstrap.cc:492] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Bootstrap starting.
I20260924 03:09:00.415941 2019 tablet_bootstrap.cc:492] T a7dc393276244199b04c546f78146659 P 21f63a2a38674f61941731b363c0d380: Bootstrap starting.
I20260924 03:09:00.416148 2020 tablet_bootstrap.cc:654] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:00.418497 2019 tablet_bootstrap.cc:654] T a7dc393276244199b04c546f78146659 P 21f63a2a38674f61941731b363c0d380: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:00.425423 2018 tablet_bootstrap.cc:492] T a7dc393276244199b04c546f78146659 P 848dacfde9424e0182b2207c8fd976af: No bootstrap required, opened a new log
I20260924 03:09:00.425719 2018 ts_tablet_manager.cc:1397] T a7dc393276244199b04c546f78146659 P 848dacfde9424e0182b2207c8fd976af: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.005s
I20260924 03:09:00.426347 2019 tablet_bootstrap.cc:492] T a7dc393276244199b04c546f78146659 P 21f63a2a38674f61941731b363c0d380: No bootstrap required, opened a new log
I20260924 03:09:00.426556 2019 ts_tablet_manager.cc:1397] T a7dc393276244199b04c546f78146659 P 21f63a2a38674f61941731b363c0d380: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260924 03:09:00.426699 2018 raft_consensus.cc:348] T a7dc393276244199b04c546f78146659 P 848dacfde9424e0182b2207c8fd976af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } }
I20260924 03:09:00.426954 2018 raft_consensus.cc:374] T a7dc393276244199b04c546f78146659 P 848dacfde9424e0182b2207c8fd976af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:00.427031 2018 raft_consensus.cc:729] T a7dc393276244199b04c546f78146659 P 848dacfde9424e0182b2207c8fd976af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 848dacfde9424e0182b2207c8fd976af, State: Initialized, Role: FOLLOWER
I20260924 03:09:00.427341 2018 consensus_queue.cc:260] T a7dc393276244199b04c546f78146659 P 848dacfde9424e0182b2207c8fd976af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } }
I20260924 03:09:00.427412 2019 raft_consensus.cc:348] T a7dc393276244199b04c546f78146659 P 21f63a2a38674f61941731b363c0d380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } }
I20260924 03:09:00.427687 2019 raft_consensus.cc:374] T a7dc393276244199b04c546f78146659 P 21f63a2a38674f61941731b363c0d380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:00.427779 2019 raft_consensus.cc:729] T a7dc393276244199b04c546f78146659 P 21f63a2a38674f61941731b363c0d380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f63a2a38674f61941731b363c0d380, State: Initialized, Role: FOLLOWER
I20260924 03:09:00.427820 2025 tablet_replica.cc:439] T a7dc393276244199b04c546f78146659 P 848dacfde9424e0182b2207c8fd976af: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } } }
I20260924 03:09:00.428231 2025 tablet_replica.cc:442] T a7dc393276244199b04c546f78146659 P 848dacfde9424e0182b2207c8fd976af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:00.428088 2019 consensus_queue.cc:260] T a7dc393276244199b04c546f78146659 P 21f63a2a38674f61941731b363c0d380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } }
I20260924 03:09:00.428576 2026 tablet_replica.cc:439] T a7dc393276244199b04c546f78146659 P 21f63a2a38674f61941731b363c0d380: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } } }
I20260924 03:09:00.428913 2026 tablet_replica.cc:442] T a7dc393276244199b04c546f78146659 P 21f63a2a38674f61941731b363c0d380: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:00.429173 2019 ts_tablet_manager.cc:1428] T a7dc393276244199b04c546f78146659 P 21f63a2a38674f61941731b363c0d380: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 03:09:00.431632 2018 ts_tablet_manager.cc:1428] T a7dc393276244199b04c546f78146659 P 848dacfde9424e0182b2207c8fd976af: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260924 03:09:00.432544 2020 tablet_bootstrap.cc:492] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: No bootstrap required, opened a new log
I20260924 03:09:00.432754 2020 ts_tablet_manager.cc:1397] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.004s
I20260924 03:09:00.433673 2020 raft_consensus.cc:348] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } }
I20260924 03:09:00.433996 2020 raft_consensus.cc:374] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:00.434087 2020 raft_consensus.cc:729] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca7f0cf6c184a4e914dd40ec2aaf2d8, State: Initialized, Role: FOLLOWER
I20260924 03:09:00.434429 2020 consensus_queue.cc:260] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } }
I20260924 03:09:00.434965 2024 tablet_replica.cc:439] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } } }
I20260924 03:09:00.435281 2024 tablet_replica.cc:442] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:00.436319 2020 ts_tablet_manager.cc:1428] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 03:09:00.553941 2024 raft_consensus.cc:482] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:00.554171 2024 raft_consensus.cc:504] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } }
I20260924 03:09:00.554847 2024 leader_election.cc:288] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 848dacfde9424e0182b2207c8fd976af (127.0.97.1:36969), 21f63a2a38674f61941731b363c0d380 (127.0.97.3:38363)
I20260924 03:09:00.555403 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7dc393276244199b04c546f78146659" candidate_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "848dacfde9424e0182b2207c8fd976af" is_pre_election: true
I20260924 03:09:00.555466 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7dc393276244199b04c546f78146659" candidate_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f63a2a38674f61941731b363c0d380" is_pre_election: true
I20260924 03:09:00.555647 1675 raft_consensus.cc:2454] T a7dc393276244199b04c546f78146659 P 848dacfde9424e0182b2207c8fd976af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ca7f0cf6c184a4e914dd40ec2aaf2d8 in term 0.
I20260924 03:09:00.555737 1954 raft_consensus.cc:2454] T a7dc393276244199b04c546f78146659 P 21f63a2a38674f61941731b363c0d380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ca7f0cf6c184a4e914dd40ec2aaf2d8 in term 0.
I20260924 03:09:00.556093 1740 leader_election.cc:302] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 848dacfde9424e0182b2207c8fd976af, 9ca7f0cf6c184a4e914dd40ec2aaf2d8; no voters:
I20260924 03:09:00.556399 2024 raft_consensus.cc:2781] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:00.556512 2024 raft_consensus.cc:482] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:00.556638 2024 raft_consensus.cc:3037] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:00.559557 2024 raft_consensus.cc:504] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } }
I20260924 03:09:00.560768 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7dc393276244199b04c546f78146659" candidate_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "848dacfde9424e0182b2207c8fd976af"
I20260924 03:09:00.560802 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7dc393276244199b04c546f78146659" candidate_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f63a2a38674f61941731b363c0d380"
I20260924 03:09:00.561010 1675 raft_consensus.cc:3037] T a7dc393276244199b04c546f78146659 P 848dacfde9424e0182b2207c8fd976af [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:00.561070 1954 raft_consensus.cc:3037] T a7dc393276244199b04c546f78146659 P 21f63a2a38674f61941731b363c0d380 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:00.564020 1675 raft_consensus.cc:2454] T a7dc393276244199b04c546f78146659 P 848dacfde9424e0182b2207c8fd976af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca7f0cf6c184a4e914dd40ec2aaf2d8 in term 1.
I20260924 03:09:00.564527 2024 leader_election.cc:288] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [CANDIDATE]: Term 1 election: Requested vote from peers 848dacfde9424e0182b2207c8fd976af (127.0.97.1:36969), 21f63a2a38674f61941731b363c0d380 (127.0.97.3:38363)
I20260924 03:09:00.564673 1740 leader_election.cc:302] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 848dacfde9424e0182b2207c8fd976af, 9ca7f0cf6c184a4e914dd40ec2aaf2d8; no voters:
I20260924 03:09:00.565403 1954 raft_consensus.cc:2454] T a7dc393276244199b04c546f78146659 P 21f63a2a38674f61941731b363c0d380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca7f0cf6c184a4e914dd40ec2aaf2d8 in term 1.
I20260924 03:09:00.566143 2024 raft_consensus.cc:2781] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:00.566380 2024 raft_consensus.cc:686] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 1 LEADER]: Becoming Leader. State: Replica: 9ca7f0cf6c184a4e914dd40ec2aaf2d8, State: Running, Role: LEADER
I20260924 03:09:00.566663 2024 consensus_queue.cc:237] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } }
I20260924 03:09:00.569171 2050 tablet_replica.cc:439] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: TxnStatusTablet state changed. Reason: New leader 9ca7f0cf6c184a4e914dd40ec2aaf2d8. Latest consensus state: current_term: 1 leader_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } } }
I20260924 03:09:00.569490 2050 tablet_replica.cc:442] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:09:00.571439 1513 catalog_manager.cc:5495] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ca7f0cf6c184a4e914dd40ec2aaf2d8 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } health_report { overall_health: UNKNOWN } } }
I20260924 03:09:00.575755 2063 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 03:09:00.576118 2063 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 03:09:00.620457 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.610375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36289 (local address 127.0.0.1:60762)
0924 03:09:00.610729 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:00.610747 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:00.610854 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:00.615311 (+ 4457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:00.615315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:00.615331 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:09:00.615586 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.615600 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.616904 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.616912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.617908 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.617919 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.618640 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.618646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.618806 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.618869 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:09:00.619874 (+ 1005us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:09:00.619894 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 03:09:00.619982 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 03:09:00.620136 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260924 03:09:00.620517 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.610388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60762 (local address 127.0.97.2:36289)
0924 03:09:00.614913 (+ 4525us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.614918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.614939 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:09:00.615022 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.615027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.615112 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.615290 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:00.615762 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.616776 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.618056 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.618481 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.618620 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.619133 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:00.619725 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:00.619911 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.620006 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.620085 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4392,"thread_start_us":4195,"threads_started":1}
I20260924 03:09:00.624904 1675 raft_consensus.cc:1261] T a7dc393276244199b04c546f78146659 P 848dacfde9424e0182b2207c8fd976af [term 1 FOLLOWER]: Refusing update from remote peer 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:00.625183 1954 raft_consensus.cc:1261] T a7dc393276244199b04c546f78146659 P 21f63a2a38674f61941731b363c0d380 [term 1 FOLLOWER]: Refusing update from remote peer 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:00.626060 2024 consensus_queue.cc:1035] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:00.626718 2024 consensus_queue.cc:1035] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:00.640426 2026 tablet_replica.cc:439] T a7dc393276244199b04c546f78146659 P 21f63a2a38674f61941731b363c0d380: TxnStatusTablet state changed. Reason: New leader 9ca7f0cf6c184a4e914dd40ec2aaf2d8. Latest consensus state: current_term: 1 leader_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } } }
I20260924 03:09:00.641395 2026 tablet_replica.cc:442] T a7dc393276244199b04c546f78146659 P 21f63a2a38674f61941731b363c0d380: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:00.645485 2024 tablet_replica.cc:439] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } } }
I20260924 03:09:00.645784 2024 tablet_replica.cc:442] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:09:00.645984 2034 mvcc.cc:205] Tried to move back new op lower bound from 7332738419191771136 to 7332738418963755008. Current Snapshot: MvccSnapshot[applied={T|T < 7332738419191771136 or (T in {7332738419191771136})}]
I20260924 03:09:00.647794 2025 tablet_replica.cc:439] T a7dc393276244199b04c546f78146659 P 848dacfde9424e0182b2207c8fd976af: TxnStatusTablet state changed. Reason: New leader 9ca7f0cf6c184a4e914dd40ec2aaf2d8. Latest consensus state: current_term: 1 leader_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } } }
I20260924 03:09:00.648105 2025 tablet_replica.cc:442] T a7dc393276244199b04c546f78146659 P 848dacfde9424e0182b2207c8fd976af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:00.649324 2026 tablet_replica.cc:439] T a7dc393276244199b04c546f78146659 P 21f63a2a38674f61941731b363c0d380: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } } }
I20260924 03:09:00.649582 2026 tablet_replica.cc:442] T a7dc393276244199b04c546f78146659 P 21f63a2a38674f61941731b363c0d380: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:00.649749 2032 mvcc.cc:205] Tried to move back new op lower bound from 7332738419191771136 to 7332738418963755008. Current Snapshot: MvccSnapshot[applied={T|T < 7332738419191771136 or (T in {7332738419191771136})}]
I20260924 03:09:00.650758 2050 tablet_replica.cc:439] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } } }
I20260924 03:09:00.650986 2050 tablet_replica.cc:442] T a7dc393276244199b04c546f78146659 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:09:00.654858 2025 tablet_replica.cc:439] T a7dc393276244199b04c546f78146659 P 848dacfde9424e0182b2207c8fd976af: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca7f0cf6c184a4e914dd40ec2aaf2d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36289 } } peers { permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 } } peers { permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 } } }
I20260924 03:09:00.655103 2025 tablet_replica.cc:442] T a7dc393276244199b04c546f78146659 P 848dacfde9424e0182b2207c8fd976af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:00.664242 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.658653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60770 (local address 127.0.97.2:36289)
0924 03:09:00.658855 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.658864 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.659349 (+ 485us) server_negotiation.cc:407] Connection header received
0924 03:09:00.659558 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.659563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.659636 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.659733 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:00.660123 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.660948 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.662353 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.662778 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.662920 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.663361 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:00.663473 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:00.663578 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.663707 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.663785 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 03:09:00.682358 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.675926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40795 (local address 127.0.97.2:36289)
0924 03:09:00.676404 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.676415 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.676865 (+ 450us) server_negotiation.cc:407] Connection header received
0924 03:09:00.677075 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.677079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.677168 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.677257 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:00.677800 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.678974 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.680439 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.680932 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.681070 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.681545 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:00.681643 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:00.681739 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.681887 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.681981 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":303}
I20260924 03:09:00.682711 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.675974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36289 (local address 127.0.97.2:40795)
0924 03:09:00.676733 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:00.676755 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:00.676954 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:00.677281 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:00.677285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:00.677295 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:09:00.677649 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.677657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.679102 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.679110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.680292 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.680303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.681120 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.681129 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.681284 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.681343 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:09:00.681744 (+ 401us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:09:00.681755 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 03:09:00.681867 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 03:09:00.682043 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":43,"thread_start_us":453,"threads_started":1}
I20260924 03:09:00.694866 1672 raft_consensus.cc:1261] T 52c4ac114fe44f92b290fb68a5b03885 P 848dacfde9424e0182b2207c8fd976af [term 1 FOLLOWER]: Refusing update from remote peer 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:00.694849 1954 raft_consensus.cc:1261] T 52c4ac114fe44f92b290fb68a5b03885 P 21f63a2a38674f61941731b363c0d380 [term 1 FOLLOWER]: Refusing update from remote peer 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:00.695611 2050 consensus_queue.cc:1035] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "848dacfde9424e0182b2207c8fd976af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:00.696084 2024 consensus_queue.cc:1035] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f63a2a38674f61941731b363c0d380" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:00.813247 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:00.813503 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:00.817185 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6500a26d98d24491be09013b0b93ebc7: Sending DeleteTablet for 3 replicas of tablet 52c4ac114fe44f92b290fb68a5b03885
I20260924 03:09:00.819332 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet 52c4ac114fe44f92b290fb68a5b03885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46802
I20260924 03:09:00.819783 1650 tablet_service.cc:1514] Processing DeleteTablet for tablet 52c4ac114fe44f92b290fb68a5b03885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59740
I20260924 03:09:00.825984 2082 tablet_replica.cc:330] T 52c4ac114fe44f92b290fb68a5b03885 P 848dacfde9424e0182b2207c8fd976af: stopping tablet replica
I20260924 03:09:00.826221 2082 raft_consensus.cc:2229] T 52c4ac114fe44f92b290fb68a5b03885 P 848dacfde9424e0182b2207c8fd976af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:00.826280 1790 tablet_service.cc:1514] Processing DeleteTablet for tablet 52c4ac114fe44f92b290fb68a5b03885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60758
I20260924 03:09:00.826457 2082 raft_consensus.cc:2258] T 52c4ac114fe44f92b290fb68a5b03885 P 848dacfde9424e0182b2207c8fd976af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:00.829097 2084 tablet_replica.cc:330] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: stopping tablet replica
I20260924 03:09:00.829144 2082 ts_tablet_manager.cc:1905] T 52c4ac114fe44f92b290fb68a5b03885 P 848dacfde9424e0182b2207c8fd976af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:00.829373 2084 raft_consensus.cc:2229] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:09:00.830271 2084 raft_consensus.cc:2258] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:00.831602 2084 ts_tablet_manager.cc:1905] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:00.832093 2081 tablet_replica.cc:330] T 52c4ac114fe44f92b290fb68a5b03885 P 21f63a2a38674f61941731b363c0d380: stopping tablet replica
I20260924 03:09:00.832296 2081 raft_consensus.cc:2229] T 52c4ac114fe44f92b290fb68a5b03885 P 21f63a2a38674f61941731b363c0d380 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:00.832584 2081 raft_consensus.cc:2258] T 52c4ac114fe44f92b290fb68a5b03885 P 21f63a2a38674f61941731b363c0d380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:00.837399 2081 ts_tablet_manager.cc:1905] T 52c4ac114fe44f92b290fb68a5b03885 P 21f63a2a38674f61941731b363c0d380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:00.841377 2082 ts_tablet_manager.cc:1918] T 52c4ac114fe44f92b290fb68a5b03885 P 848dacfde9424e0182b2207c8fd976af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:09:00.841540 2082 log.cc:1196] T 52c4ac114fe44f92b290fb68a5b03885 P 848dacfde9424e0182b2207c8fd976af: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/52c4ac114fe44f92b290fb68a5b03885
I20260924 03:09:00.841979 2082 ts_tablet_manager.cc:1939] T 52c4ac114fe44f92b290fb68a5b03885 P 848dacfde9424e0182b2207c8fd976af: Deleting consensus metadata
I20260924 03:09:00.843241 1497 catalog_manager.cc:4841] TS 848dacfde9424e0182b2207c8fd976af (127.0.97.1:36969): tablet 52c4ac114fe44f92b290fb68a5b03885 (table test-table [id=f2563cff2a494f24a8a357a1ace608c8]) successfully deleted
I20260924 03:09:00.846484 2084 ts_tablet_manager.cc:1918] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:09:00.846644 2084 log.cc:1196] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/52c4ac114fe44f92b290fb68a5b03885
I20260924 03:09:00.847126 2084 ts_tablet_manager.cc:1939] T 52c4ac114fe44f92b290fb68a5b03885 P 9ca7f0cf6c184a4e914dd40ec2aaf2d8: Deleting consensus metadata
I20260924 03:09:00.848453 2081 ts_tablet_manager.cc:1918] T 52c4ac114fe44f92b290fb68a5b03885 P 21f63a2a38674f61941731b363c0d380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:09:00.848626 2081 log.cc:1196] T 52c4ac114fe44f92b290fb68a5b03885 P 21f63a2a38674f61941731b363c0d380: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/52c4ac114fe44f92b290fb68a5b03885
I20260924 03:09:00.848841 1499 catalog_manager.cc:4841] TS 9ca7f0cf6c184a4e914dd40ec2aaf2d8 (127.0.97.2:36289): tablet 52c4ac114fe44f92b290fb68a5b03885 (table test-table [id=f2563cff2a494f24a8a357a1ace608c8]) successfully deleted
I20260924 03:09:00.849092 2081 ts_tablet_manager.cc:1939] T 52c4ac114fe44f92b290fb68a5b03885 P 21f63a2a38674f61941731b363c0d380: Deleting consensus metadata
I20260924 03:09:00.850219 1499 catalog_manager.cc:4841] TS 21f63a2a38674f61941731b363c0d380 (127.0.97.3:38363): tablet 52c4ac114fe44f92b290fb68a5b03885 (table test-table [id=f2563cff2a494f24a8a357a1ace608c8]) successfully deleted
Sep 24 03:09:00 dist-test-slave-5l98 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219339, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:09:00.858629 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.832813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59744 (local address 127.0.97.1:36969)
0924 03:09:00.836945 (+ 4132us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.836955 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.839198 (+ 2243us) server_negotiation.cc:407] Connection header received
0924 03:09:00.839272 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.839277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.839359 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.839467 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:00.842294 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.843608 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.845576 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.846293 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.846404 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.852431 (+ 6027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:00.852447 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:00.852464 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:00.852555 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:00.855589 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.856410 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.856418 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.856420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.856628 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.857026 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.857031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.857034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.857407 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:00.857719 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.858004 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.858226 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3939,"thread_start_us":3826,"threads_started":1}
W20260924 03:09:00.859356 1715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59744
I20260924 03:09:00.878297 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.862837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39574 (local address 127.0.97.62:40325)
0924 03:09:00.863071 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.863077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.863280 (+ 203us) server_negotiation.cc:407] Connection header received
0924 03:09:00.863421 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.863428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.863510 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.863621 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:00.865069 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.866299 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.867723 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.868366 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.868520 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.871208 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:00.871216 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:00.871221 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:00.871318 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:00.874758 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.875481 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.875487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.875490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.875634 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.876139 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.876145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.876148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.876595 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:00.876865 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.877145 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.877425 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260924 03:09:00.879079 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39574
I20260924 03:09:00.882879 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 1584
I20260924 03:09:00.899796 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.884132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59754 (local address 127.0.97.1:36969)
0924 03:09:00.884351 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.884357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.884673 (+ 316us) server_negotiation.cc:407] Connection header received
0924 03:09:00.884850 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.884853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.884948 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.885033 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:00.886608 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.887372 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.889208 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.889620 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.889730 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.892778 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:00.892786 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:00.892791 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:00.892836 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:00.896591 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.897462 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.897468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.897472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.897631 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.898236 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.898244 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.898249 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.898608 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:00.898863 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.899229 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.899463 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 03:09:00.900417 1715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59754
W20260924 03:09:00.901185 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 1584: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:09:00.901410 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 1584
I20260924 03:09:00.919185 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 1724
Sep 24 03:09:00 dist-test-slave-5l98 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219339, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 03:09:00.936105 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.919988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60780 (local address 127.0.97.2:36289)
0924 03:09:00.920174 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.920180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.920202 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:09:00.920304 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.920310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.920390 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.920541 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:00.921651 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.922795 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.923965 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.924727 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.924850 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.929798 (+ 4948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:00.929812 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:00.929832 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:00.929867 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:00.933233 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.933982 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.933993 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.934001 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.934119 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.934534 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.934539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.934544 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.934927 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:00.935254 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.935546 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.935760 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 03:09:00.936987 1855 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60780
W20260924 03:09:00.937656 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 1724: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:09:00.937824 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 1724
I20260924 03:09:00.957540 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 1864
Sep 24 03:09:00 dist-test-slave-5l98 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219339, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 03:09:00.973946 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.958361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46830 (local address 127.0.97.3:38363)
0924 03:09:00.958833 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.958838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.958855 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:09:00.958910 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.958914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.958979 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.959069 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:00.960313 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.961091 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.962338 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.962765 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.962869 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.967593 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:00.967607 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:00.967623 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:00.967661 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:00.971023 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.971807 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.971815 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.971817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.971957 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.972338 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.972343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.972345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.972805 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:00.973067 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.973357 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.973528 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":254,"threads_started":1}
W20260924 03:09:00.974562 1994 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46830
W20260924 03:09:00.975168 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 1864: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:09:00.975337 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 1864
I20260924 03:09:00.997582 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 1481
I20260924 03:09:00.998772 1553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:01.122499 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 1481
2026-09-24T03:09:01Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5166 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:09:01 dist-test-slave-5l98 krb5kdc[2094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:09:01 dist-test-slave-5l98 krb5kdc[2094](info): set up 2 sockets
Sep 24 03:09:01 dist-test-slave-5l98 krb5kdc[2094](info): commencing operation
krb5kdc: starting...
W20260924 03:09:02.241235 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:09:02 dist-test-slave-5l98 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:09:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:09:02.454067 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37099
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:02.779330 2107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:02.779641 2107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:02.779807 2107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:02.787189 2107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:02.787304 2107 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 03:09:02.787379 2107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:02.787472 2107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:02.787561 2107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:02.787672 2107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:09:02.797911 2107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37099
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:02.799371 2107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:02.800872 2107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:02.813089 2112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:02.813119 2113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:02.813177 2115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:02.814159 2107 server_base.cc:1034] running on GCE node
I20260924 03:09:02.815177 2107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:02.816783 2107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:02.818037 2107 hybrid_clock.cc:648] HybridClock initialized: now 1790219342817969 us; error 45 us; skew 500 ppm
I20260924 03:09:02.818382 2107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:02 dist-test-slave-5l98 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:02.826519 2107 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:09:02.828161 2107 webserver.cc:460] Webserver started at http://127.0.97.62:38309/ using document root <none> and password file <none>
I20260924 03:09:02.829066 2107 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:02.829209 2107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:02.829535 2107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:02.832599 2107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "ffa7ec1169d7481cb599370d0f9f62f9"
format_stamp: "Formatted at 2026-09-24 03:09:02 on dist-test-slave-5l98"
server_key: "fe10da4996c8bd850da415ea9cd03de1"
server_key_iv: "16655d73b52b90477f112294071c32a6"
server_key_version: "encryptionkey@0"
I20260924 03:09:02.833398 2107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "ffa7ec1169d7481cb599370d0f9f62f9"
format_stamp: "Formatted at 2026-09-24 03:09:02 on dist-test-slave-5l98"
server_key: "fe10da4996c8bd850da415ea9cd03de1"
server_key_iv: "16655d73b52b90477f112294071c32a6"
server_key_version: "encryptionkey@0"
I20260924 03:09:02.839221 2107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 03:09:02.843389 2122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:02.845104 2107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:09:02.845336 2107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "ffa7ec1169d7481cb599370d0f9f62f9"
format_stamp: "Formatted at 2026-09-24 03:09:02 on dist-test-slave-5l98"
server_key: "fe10da4996c8bd850da415ea9cd03de1"
server_key_iv: "16655d73b52b90477f112294071c32a6"
server_key_version: "encryptionkey@0"
I20260924 03:09:02.845572 2107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:02.859396 2107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:02.863969 2107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:02.864243 2107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:02.886543 2107 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37099
I20260924 03:09:02.886551 2183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37099 every 8 connection(s)
I20260924 03:09:02.888175 2107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:09:02.892115 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:02.893352 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 2107
I20260924 03:09:02.893555 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:09:02.894362 388 external_mini_cluster.cc:1428] Setting key d43af063bce297af278e3fc0b6fa17cb
I20260924 03:09:02.907406 2184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9: Bootstrap starting.
Sep 24 03:09:02 dist-test-slave-5l98 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:02.911319 2184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:02.912400 2184 log.cc:824] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:02.915355 2184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9: No bootstrap required, opened a new log
I20260924 03:09:02.918771 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:02.896637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58652 (local address 127.0.97.62:37099)
0924 03:09:02.897576 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:02.897589 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:02.897636 (+ 47us) server_negotiation.cc:407] Connection header received
0924 03:09:02.899247 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:02.899273 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:02.899802 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:02.900248 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:02.902160 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:02.903560 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:02.905445 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:02.906308 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:02.906448 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:02.911234 (+ 4786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:02.911243 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:02.911260 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:02.911321 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:02.914603 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:02.915359 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:02.915366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:02.915372 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:02.915581 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:02.915967 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:02.915972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:02.915975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:02.916550 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:02.916871 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:02.917288 (+ 417us) server_negotiation.cc:299] Negotiation successful
0924 03:09:02.917728 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":448,"threads_started":1}
I20260924 03:09:02.923225 2184 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa7ec1169d7481cb599370d0f9f62f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37099 } }
I20260924 03:09:02.923579 2184 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:02.923682 2184 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa7ec1169d7481cb599370d0f9f62f9, State: Initialized, Role: FOLLOWER
I20260924 03:09:02.924256 2184 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa7ec1169d7481cb599370d0f9f62f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37099 } }
I20260924 03:09:02.924479 2184 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:09:02.924639 2184 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:09:02.924799 2184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:02.927019 2184 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa7ec1169d7481cb599370d0f9f62f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37099 } }
I20260924 03:09:02.927526 2184 leader_election.cc:302] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffa7ec1169d7481cb599370d0f9f62f9; no voters:
I20260924 03:09:02.928089 2184 leader_election.cc:288] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:09:02.928224 2189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:02.929528 2189 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9 [term 1 LEADER]: Becoming Leader. State: Replica: ffa7ec1169d7481cb599370d0f9f62f9, State: Running, Role: LEADER
I20260924 03:09:02.930055 2189 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa7ec1169d7481cb599370d0f9f62f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37099 } }
I20260924 03:09:02.930393 2184 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:09:02.932978 2191 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffa7ec1169d7481cb599370d0f9f62f9. Latest consensus state: current_term: 1 leader_uuid: "ffa7ec1169d7481cb599370d0f9f62f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa7ec1169d7481cb599370d0f9f62f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37099 } } }
I20260924 03:09:02.933336 2191 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:02.933528 2190 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffa7ec1169d7481cb599370d0f9f62f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa7ec1169d7481cb599370d0f9f62f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37099 } } }
I20260924 03:09:02.933830 2190 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:02.935796 2197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:09:02.941761 2197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:09:02.951901 2197 catalog_manager.cc:1347] Generated new cluster ID: 6ba0253404f94e2f80ce416ba65a0c53
I20260924 03:09:02.952064 2197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:09:02.974920 2197 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:09:02.976387 2197 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:09:02.984805 2197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9: Generated new TSK 0
I20260924 03:09:02.986143 2197 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 03:09:03.041383 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37099
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:03.377537 2210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:03.377831 2210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:03.378213 2210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:03.385799 2210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:03.385927 2210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:03.386068 2210 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 03:09:03.386144 2210 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:09:03.386286 2210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:09:03.396176 2210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:03.397557 2210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:03.398881 2210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:03.413519 2215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:03.413590 2216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:03.413704 2218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:03.414582 2210 server_base.cc:1034] running on GCE node
I20260924 03:09:03.415185 2210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:03.416147 2210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:03.417367 2210 hybrid_clock.cc:648] HybridClock initialized: now 1790219343417319 us; error 41 us; skew 500 ppm
I20260924 03:09:03.417784 2210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:03 dist-test-slave-5l98 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:03.421289 2210 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:09:03.422721 2210 webserver.cc:460] Webserver started at http://127.0.97.1:46317/ using document root <none> and password file <none>
I20260924 03:09:03.423386 2210 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:03.423489 2210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:03.423758 2210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:03.426723 2210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88"
format_stamp: "Formatted at 2026-09-24 03:09:03 on dist-test-slave-5l98"
server_key: "3fa169f7ff2cd8fae297f1652a6552db"
server_key_iv: "10bb87ea49966f569c2d97173901bce9"
server_key_version: "encryptionkey@0"
I20260924 03:09:03.427462 2210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88"
format_stamp: "Formatted at 2026-09-24 03:09:03 on dist-test-slave-5l98"
server_key: "3fa169f7ff2cd8fae297f1652a6552db"
server_key_iv: "10bb87ea49966f569c2d97173901bce9"
server_key_version: "encryptionkey@0"
I20260924 03:09:03.432741 2210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 03:09:03.436430 2225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:03.437914 2210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:09:03.438097 2210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88"
format_stamp: "Formatted at 2026-09-24 03:09:03 on dist-test-slave-5l98"
server_key: "3fa169f7ff2cd8fae297f1652a6552db"
server_key_iv: "10bb87ea49966f569c2d97173901bce9"
server_key_version: "encryptionkey@0"
I20260924 03:09:03.438320 2210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:03.467437 2210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:03.472370 2210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:03.472736 2210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:03.475787 2210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:03.475962 2210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:03.476116 2210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:03.476222 2210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:09:03 dist-test-slave-5l98 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:03.500690 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:03.476384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42471 (local address 127.0.97.62:37099)
0924 03:09:03.477106 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:03.477113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:03.478229 (+ 1116us) server_negotiation.cc:407] Connection header received
0924 03:09:03.480412 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:03.480417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:03.480533 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:03.480662 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:03.483095 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:03.484623 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:03.487355 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:03.488283 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:03.488472 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:03.493572 (+ 5100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:03.493577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:03.493580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:03.493639 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:03.497145 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:03.498211 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:03.498214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:03.498217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:03.498391 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:03.498943 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:03.498948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:03.498952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:03.499227 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:03.499383 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:03.500085 (+ 702us) server_negotiation.cc:299] Negotiation successful
0924 03:09:03.500293 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":493,"threads_started":1}
I20260924 03:09:03.502012 2235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:03.477132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37099 (local address 127.0.97.1:42471)
0924 03:09:03.478035 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:03.478084 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:03.480127 (+ 2043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:03.480916 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:03.480929 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:03.481755 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:03.482795 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:03.482820 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:03.484798 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:03.484824 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:03.487149 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:03.487180 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:03.488708 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:03.488718 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:03.488953 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:03.489907 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:03.489931 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:03.493300 (+ 3369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:03.497354 (+ 4054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:03.497361 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:03.497377 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:03.498047 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:03.498539 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:03.498543 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:03.498549 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:03.498780 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:03.499370 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:03.499378 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:03.499722 (+ 344us) client_negotiation.cc:769] Sending connection context
0924 03:09:03.500001 (+ 279us) client_negotiation.cc:240] Negotiation successful
0924 03:09:03.500554 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":409,"threads_started":1}
I20260924 03:09:03.524061 2294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:03.508008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37099 (local address 127.0.97.1:41187)
0924 03:09:03.509024 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:03.509044 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:03.509215 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:03.509669 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:03.509678 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:03.510670 (+ 992us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:03.511170 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:03.511197 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:03.512529 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:03.512539 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:03.513571 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:03.513582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:03.514351 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:03.514357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:03.514543 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:03.515396 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:03.515422 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:03.517023 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:03.520349 (+ 3326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:03.520355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:03.520359 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:03.520910 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:03.521457 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:03.521464 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:03.521469 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:03.521659 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:03.522447 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:03.522452 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:03.522784 (+ 332us) client_negotiation.cc:769] Sending connection context
0924 03:09:03.522965 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 03:09:03.523319 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"thread_start_us":714,"threads_started":1}
I20260924 03:09:03.524580 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:03.508072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41187 (local address 127.0.97.62:37099)
0924 03:09:03.508310 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:03.508319 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:03.509139 (+ 820us) server_negotiation.cc:407] Connection header received
0924 03:09:03.509360 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:03.509366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:03.509455 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:03.509561 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:03.511382 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:03.512378 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:03.513715 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:03.514151 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:03.514298 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:03.517238 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:03.517243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:03.517247 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:03.517284 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:03.520166 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:03.521105 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:03.521113 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:03.521118 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:03.521292 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:03.521784 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:03.521791 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:03.521794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:03.522177 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:03.522337 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:03.523940 (+ 1603us) server_negotiation.cc:299] Negotiation successful
0924 03:09:03.524176 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:09:03.536787 2210 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42511
I20260924 03:09:03.536907 2345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42511 every 8 connection(s)
I20260924 03:09:03.538129 2210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:09:03.540382 2346 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37099
I20260924 03:09:03.540639 2346 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:03.541425 2346 heartbeater.cc:511] Master 127.0.97.62:37099 requested a full tablet report, sending...
I20260924 03:09:03.543213 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 2210
I20260924 03:09:03.543444 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:09:03.543885 2139 ts_manager.cc:194] Registered new tserver with Master: 1c77b6c7f2fd44fcb8e81719b50e9b88 (127.0.97.1:42511)
I20260924 03:09:03.544282 388 external_mini_cluster.cc:1428] Setting key 158b43ddd506f2d0c8bddb4f004f78f1
I20260924 03:09:03.546000 2139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42471
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 03:09:03.588882 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37099
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:03.921234 2350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:03.921530 2350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:03.921711 2350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:03.929471 2350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:03.929592 2350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:03.929738 2350 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 03:09:03.929826 2350 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:09:03.929965 2350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:09:03.940064 2350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:03.941447 2350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:03.942914 2350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:03.956461 2355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:03.956434 2356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:03.956431 2358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:03.957480 2350 server_base.cc:1034] running on GCE node
I20260924 03:09:03.958163 2350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:03.959151 2350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:03.960371 2350 hybrid_clock.cc:648] HybridClock initialized: now 1790219343960320 us; error 44 us; skew 500 ppm
I20260924 03:09:03.960750 2350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:03 dist-test-slave-5l98 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:03.963722 2350 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:09:03.965193 2350 webserver.cc:460] Webserver started at http://127.0.97.2:36789/ using document root <none> and password file <none>
I20260924 03:09:03.965876 2350 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:03.965978 2350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:03.966231 2350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:03.969146 2350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "b3b67e6f90164055b29ca57d13548479"
format_stamp: "Formatted at 2026-09-24 03:09:03 on dist-test-slave-5l98"
server_key: "1ddd1e3688c19fbfb5d29158bd5cd1bb"
server_key_iv: "5288736fa5be07c96d87ce1b7b0f5410"
server_key_version: "encryptionkey@0"
I20260924 03:09:03.970074 2350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b3b67e6f90164055b29ca57d13548479"
format_stamp: "Formatted at 2026-09-24 03:09:03 on dist-test-slave-5l98"
server_key: "1ddd1e3688c19fbfb5d29158bd5cd1bb"
server_key_iv: "5288736fa5be07c96d87ce1b7b0f5410"
server_key_version: "encryptionkey@0"
I20260924 03:09:03.975193 2350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:09:03.978806 2365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:03.980294 2350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:09:03.980479 2350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "b3b67e6f90164055b29ca57d13548479"
format_stamp: "Formatted at 2026-09-24 03:09:03 on dist-test-slave-5l98"
server_key: "1ddd1e3688c19fbfb5d29158bd5cd1bb"
server_key_iv: "5288736fa5be07c96d87ce1b7b0f5410"
server_key_version: "encryptionkey@0"
I20260924 03:09:03.980700 2350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:04.008925 2350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:04.013507 2350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:04.013865 2350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:04.016578 2350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:04.016706 2350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:04.016831 2350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:04.016918 2350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:09:04 dist-test-slave-5l98 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:04.042239 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.017398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42861 (local address 127.0.97.62:37099)
0924 03:09:04.017577 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:04.017584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:04.019089 (+ 1505us) server_negotiation.cc:407] Connection header received
0924 03:09:04.021524 (+ 2435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:04.021530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:04.021619 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:04.021748 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:04.024107 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.025442 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.027971 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.028656 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.028815 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.033986 (+ 5171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:04.033995 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:04.034001 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:04.034051 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:04.037687 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:04.038706 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:04.038713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:04.038717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:04.038846 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:04.039489 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:04.039492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:04.039494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:04.039901 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:04.040045 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:04.041332 (+ 1287us) server_negotiation.cc:299] Negotiation successful
0924 03:09:04.041657 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 03:09:04.043246 2374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.017960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37099 (local address 127.0.97.2:42861)
0924 03:09:04.018895 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:04.018946 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:04.021181 (+ 2235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:04.022030 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:04.022046 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:04.022918 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:04.023878 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.023915 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.025635 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.025642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.027789 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.027802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.029071 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.029076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.029265 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.030091 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:04.030132 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:04.033709 (+ 3577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:04.037834 (+ 4125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:04.037847 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:04.037873 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:04.038559 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:04.038988 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:04.038992 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:04.038994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:04.039328 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:04.040113 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:04.040119 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:04.040643 (+ 524us) client_negotiation.cc:769] Sending connection context
0924 03:09:04.041239 (+ 596us) client_negotiation.cc:240] Negotiation successful
0924 03:09:04.041811 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":424,"threads_started":1}
I20260924 03:09:04.067301 2445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.050671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37099 (local address 127.0.97.2:38877)
0924 03:09:04.051532 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:04.051557 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:04.051801 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:04.052199 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:04.052203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:04.052802 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:04.053363 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.053375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.054483 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.054490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.055620 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.055631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.056528 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.056550 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.056712 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.057559 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:04.057589 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:04.059097 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:04.063933 (+ 4836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:04.063945 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:04.063949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:04.064489 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:04.065051 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:04.065054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:04.065056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:04.065181 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:04.066126 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:04.066130 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:04.066361 (+ 231us) client_negotiation.cc:769] Sending connection context
0924 03:09:04.066503 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 03:09:04.066798 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":385,"threads_started":1}
I20260924 03:09:04.067477 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.050671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38877 (local address 127.0.97.62:37099)
0924 03:09:04.050875 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:04.050884 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:04.051664 (+ 780us) server_negotiation.cc:407] Connection header received
0924 03:09:04.051971 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:04.051977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:04.052060 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:04.052176 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:04.053532 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.054339 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.055770 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.056264 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.056428 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.059366 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:04.059375 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:04.059382 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:04.059433 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:04.063619 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:04.064676 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:04.064686 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:04.064693 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:04.064904 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:04.065307 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:04.065315 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:04.065321 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:04.065903 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:04.066106 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:04.066679 (+ 573us) server_negotiation.cc:299] Negotiation successful
0924 03:09:04.067014 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:09:04.075573 2350 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44791
I20260924 03:09:04.075590 2484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44791 every 8 connection(s)
I20260924 03:09:04.076876 2350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:09:04.077621 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 2350
I20260924 03:09:04.077807 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:09:04.078362 388 external_mini_cluster.cc:1428] Setting key 37f7341ca2ebb5959ff8bb729776fb91
I20260924 03:09:04.079213 2485 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37099
I20260924 03:09:04.079511 2485 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:04.080243 2485 heartbeater.cc:511] Master 127.0.97.62:37099 requested a full tablet report, sending...
I20260924 03:09:04.081593 2139 ts_manager.cc:194] Registered new tserver with Master: b3b67e6f90164055b29ca57d13548479 (127.0.97.2:44791)
I20260924 03:09:04.082422 2139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42861
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 03:09:04.125711 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37099
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:04.466766 2489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:04.467116 2489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:04.467290 2489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:04.474808 2489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:04.474968 2489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:04.475126 2489 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 03:09:04.475214 2489 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:09:04.475356 2489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:09:04.485503 2489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:04.486865 2489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:04.488219 2489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:04.502272 2494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:04.502215 2495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:04.502345 2497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:04.503055 2489 server_base.cc:1034] running on GCE node
I20260924 03:09:04.503743 2489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:04.504853 2489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:04.506068 2489 hybrid_clock.cc:648] HybridClock initialized: now 1790219344505985 us; error 78 us; skew 500 ppm
I20260924 03:09:04.506417 2489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:04 dist-test-slave-5l98 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:04.510022 2489 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:09:04.511651 2489 webserver.cc:460] Webserver started at http://127.0.97.3:36789/ using document root <none> and password file <none>
I20260924 03:09:04.512349 2489 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:04.512562 2489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:04.512953 2489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:04.515875 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "ff49ff7c1f3c46deb3d651443b6f8303"
format_stamp: "Formatted at 2026-09-24 03:09:04 on dist-test-slave-5l98"
server_key: "1aa48342637da32af53e53ee9c555874"
server_key_iv: "31cfdf4703c6dc8bead88a499af1a260"
server_key_version: "encryptionkey@0"
I20260924 03:09:04.516762 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ff49ff7c1f3c46deb3d651443b6f8303"
format_stamp: "Formatted at 2026-09-24 03:09:04 on dist-test-slave-5l98"
server_key: "1aa48342637da32af53e53ee9c555874"
server_key_iv: "31cfdf4703c6dc8bead88a499af1a260"
server_key_version: "encryptionkey@0"
I20260924 03:09:04.522104 2489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 03:09:04.526216 2504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:04.527951 2489 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 03:09:04.528156 2489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "ff49ff7c1f3c46deb3d651443b6f8303"
format_stamp: "Formatted at 2026-09-24 03:09:04 on dist-test-slave-5l98"
server_key: "1aa48342637da32af53e53ee9c555874"
server_key_iv: "31cfdf4703c6dc8bead88a499af1a260"
server_key_version: "encryptionkey@0"
I20260924 03:09:04.528383 2489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:04.548308 2489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:04.549707 2346 heartbeater.cc:503] Master 127.0.97.62:37099 was elected leader, sending a full tablet report...
I20260924 03:09:04.553678 2489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:04.553953 2489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:04.557169 2489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:04.557296 2489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:04.557423 2489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:04.557521 2489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:09:04 dist-test-slave-5l98 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:04.582932 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.557595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55923 (local address 127.0.97.62:37099)
0924 03:09:04.557773 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:04.557780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:04.559188 (+ 1408us) server_negotiation.cc:407] Connection header received
0924 03:09:04.561289 (+ 2101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:04.561297 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:04.561417 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:04.561536 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:04.563576 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.564572 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.567211 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.567856 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.568004 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.574520 (+ 6516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:04.574530 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:04.574534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:04.574582 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:04.578799 (+ 4217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:04.580161 (+ 1362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:04.580166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:04.580168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:04.580306 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:04.581004 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:04.581011 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:04.581016 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:04.581399 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:04.581593 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:04.582324 (+ 731us) server_negotiation.cc:299] Negotiation successful
0924 03:09:04.582553 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 03:09:04.584121 2513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.558139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37099 (local address 127.0.97.3:55923)
0924 03:09:04.559001 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:04.559047 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:04.561018 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:04.561795 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:04.561807 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:04.562550 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:04.563392 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.563413 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.564737 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.564746 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.567021 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.567038 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.568167 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.568179 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.568532 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.569522 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:04.569544 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:04.574096 (+ 4552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:04.578980 (+ 4884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:04.578996 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:04.579025 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:04.579952 (+ 927us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:04.580447 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:04.580456 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:04.580463 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:04.580838 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:04.581538 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:04.581550 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:04.581954 (+ 404us) client_negotiation.cc:769] Sending connection context
0924 03:09:04.582226 (+ 272us) client_negotiation.cc:240] Negotiation successful
0924 03:09:04.582788 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":375,"threads_started":1}
I20260924 03:09:04.605310 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.590209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53893 (local address 127.0.97.62:37099)
0924 03:09:04.590368 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:04.590372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:04.591180 (+ 808us) server_negotiation.cc:407] Connection header received
0924 03:09:04.591389 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:04.591394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:04.591462 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:04.591563 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:04.592456 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.593334 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.595414 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.596012 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.596187 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.598882 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:04.598888 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:04.598896 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:04.598940 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:04.602304 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:04.603033 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:04.603041 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:04.603046 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:04.603174 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:04.603705 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:04.603710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:04.603714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:04.603995 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:04.604187 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:04.604646 (+ 459us) server_negotiation.cc:299] Negotiation successful
0924 03:09:04.604978 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 03:09:04.605369 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.590333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37099 (local address 127.0.97.3:53893)
0924 03:09:04.591070 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:04.591091 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:04.591253 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:04.591586 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:04.591590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:04.592048 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:04.592313 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.592321 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.593507 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.593512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.595261 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.595277 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.596248 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.596256 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.596437 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.597248 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:04.597265 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:04.598584 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:04.602477 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:04.602482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:04.602488 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:04.602911 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:04.603398 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:04.603404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:04.603410 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:04.603591 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:04.604115 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:04.604120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:04.604316 (+ 196us) client_negotiation.cc:769] Sending connection context
0924 03:09:04.604466 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 03:09:04.604885 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":395,"threads_started":1}
I20260924 03:09:04.617828 2489 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40409
I20260924 03:09:04.617863 2623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40409 every 8 connection(s)
I20260924 03:09:04.619125 2489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:09:04.621551 2624 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37099
I20260924 03:09:04.621774 2624 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:04.622442 2624 heartbeater.cc:511] Master 127.0.97.62:37099 requested a full tablet report, sending...
I20260924 03:09:04.623849 2139 ts_manager.cc:194] Registered new tserver with Master: ff49ff7c1f3c46deb3d651443b6f8303 (127.0.97.3:40409)
I20260924 03:09:04.624804 2139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55923
I20260924 03:09:04.625666 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 2489
I20260924 03:09:04.625959 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:09:04.626433 388 external_mini_cluster.cc:1428] Setting key 308ea96849578900df1479c4b67f725e
I20260924 03:09:04.629622 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:09:04 dist-test-slave-5l98 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:09:04 dist-test-slave-5l98 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:04.679256 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.663199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58662 (local address 127.0.97.62:37099)
0924 03:09:04.663394 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:04.663399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:04.663658 (+ 259us) server_negotiation.cc:407] Connection header received
0924 03:09:04.663889 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:04.663893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:04.663957 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:04.664052 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:04.665663 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.666448 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.667768 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.668218 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.668335 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.673463 (+ 5128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:04.673469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:04.673472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:04.673533 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:04.676629 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:04.677392 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:04.677398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:04.677400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:04.677532 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:04.677956 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:04.677959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:04.677961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:04.678279 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:04.678499 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:04.678791 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 03:09:04.678977 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 03:09:04.683775 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:09:04.691028 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:04.717947 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.703549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40409 (local address 127.0.0.1:59700)
0924 03:09:04.705353 (+ 1804us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:04.705402 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:04.705631 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:04.706472 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:04.706481 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:04.706514 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:04.707055 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.707069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.711126 (+ 4057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.711143 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.714279 (+ 3136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.714307 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.716886 (+ 2579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.716892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.717109 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.717163 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 03:09:04.717320 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 03:09:04.717539 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1557,"thread_start_us":1406,"threads_started":1}
I20260924 03:09:04.719184 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.703743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59700 (local address 127.0.97.3:40409)
0924 03:09:04.704901 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:04.704911 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:04.705526 (+ 615us) server_negotiation.cc:407] Connection header received
0924 03:09:04.705890 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:04.705900 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:04.706186 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:04.706458 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:04.707680 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.710839 (+ 3159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.714492 (+ 3653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.716243 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.716374 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.716677 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:04.718598 (+ 1921us) server_negotiation.cc:299] Negotiation successful
0924 03:09:04.718809 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":949,"thread_start_us":690,"threads_started":1}
I20260924 03:09:04.719447 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.705943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44791 (local address 127.0.0.1:55266)
0924 03:09:04.709601 (+ 3658us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:04.709631 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:04.710002 (+ 371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:04.711112 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:04.711117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:04.711152 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:04.711721 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.711734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.713900 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.713905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.716327 (+ 2422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.716341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.718583 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.718589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.718769 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.718816 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 03:09:04.718911 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 03:09:04.719058 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3498,"thread_start_us":3288,"threads_started":1}
I20260924 03:09:04.720005 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.706287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55266 (local address 127.0.97.2:44791)
0924 03:09:04.707267 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:04.707276 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:04.710179 (+ 2903us) server_negotiation.cc:407] Connection header received
0924 03:09:04.710339 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:04.710348 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:04.710630 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:04.710855 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:04.712055 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.713712 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.716543 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.718076 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.718208 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.718442 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:04.719668 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0924 03:09:04.719793 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":584,"threads_started":1}
I20260924 03:09:04.725378 2415 tablet_service.cc:1467] Processing CreateTablet for tablet 18041d4cf18547b7b52daa36ea54cda5 (DEFAULT_TABLE table=test-table [id=d5b3fd7228104af5bd5d786563313378]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:04.725206 2554 tablet_service.cc:1467] Processing CreateTablet for tablet 18041d4cf18547b7b52daa36ea54cda5 (DEFAULT_TABLE table=test-table [id=d5b3fd7228104af5bd5d786563313378]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:04.726686 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18041d4cf18547b7b52daa36ea54cda5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:04.726742 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18041d4cf18547b7b52daa36ea54cda5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:04.728971 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.710271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42511 (local address 127.0.0.1:35090)
0924 03:09:04.718284 (+ 8013us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:04.718316 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:04.718416 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:04.721507 (+ 3091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:04.721512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:04.721542 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:04.721720 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.721730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.723867 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.723873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.725939 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.725956 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.728009 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.728016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.728235 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.728272 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 03:09:04.728392 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 03:09:04.728623 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7861,"thread_start_us":10365,"threads_started":1}
I20260924 03:09:04.729712 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.709911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.0.97.1:42511)
0924 03:09:04.720809 (+ 10898us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:04.720821 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:04.720853 (+ 32us) server_negotiation.cc:407] Connection header received
0924 03:09:04.720948 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:04.720954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:04.721278 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:04.721498 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:04.722061 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.723668 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.726131 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.727795 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.727955 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.728177 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:04.728465 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 03:09:04.728688 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10688,"thread_start_us":10517,"threads_started":1}
I20260924 03:09:04.734654 2276 tablet_service.cc:1467] Processing CreateTablet for tablet 18041d4cf18547b7b52daa36ea54cda5 (DEFAULT_TABLE table=test-table [id=d5b3fd7228104af5bd5d786563313378]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:04.736263 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18041d4cf18547b7b52daa36ea54cda5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:04.741355 2643 tablet_bootstrap.cc:492] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303: Bootstrap starting.
I20260924 03:09:04.742786 2644 tablet_bootstrap.cc:492] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479: Bootstrap starting.
I20260924 03:09:04.746047 2643 tablet_bootstrap.cc:654] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:04.746673 2644 tablet_bootstrap.cc:654] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:04.747666 2643 log.cc:824] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:04.747956 2644 log.cc:824] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:04.750941 2643 tablet_bootstrap.cc:492] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303: No bootstrap required, opened a new log
I20260924 03:09:04.751240 2643 ts_tablet_manager.cc:1397] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 03:09:04.751334 2644 tablet_bootstrap.cc:492] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479: No bootstrap required, opened a new log
I20260924 03:09:04.751694 2644 ts_tablet_manager.cc:1397] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 03:09:04.753255 2647 tablet_bootstrap.cc:492] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Bootstrap starting.
I20260924 03:09:04.757392 2647 tablet_bootstrap.cc:654] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:04.759024 2647 log.cc:824] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:04.762079 2644 raft_consensus.cc:348] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } }
I20260924 03:09:04.762512 2644 raft_consensus.cc:374] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:04.762624 2644 raft_consensus.cc:729] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b67e6f90164055b29ca57d13548479, State: Initialized, Role: FOLLOWER
I20260924 03:09:04.763288 2644 consensus_queue.cc:260] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } }
I20260924 03:09:04.763144 2643 raft_consensus.cc:348] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } }
I20260924 03:09:04.763567 2643 raft_consensus.cc:374] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:04.763672 2643 raft_consensus.cc:729] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff49ff7c1f3c46deb3d651443b6f8303, State: Initialized, Role: FOLLOWER
I20260924 03:09:04.763705 2647 tablet_bootstrap.cc:492] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88: No bootstrap required, opened a new log
I20260924 03:09:04.763993 2647 ts_tablet_manager.cc:1397] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20260924 03:09:04.764542 2485 heartbeater.cc:503] Master 127.0.97.62:37099 was elected leader, sending a full tablet report...
I20260924 03:09:04.764384 2643 consensus_queue.cc:260] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } }
I20260924 03:09:04.765509 2624 heartbeater.cc:503] Master 127.0.97.62:37099 was elected leader, sending a full tablet report...
I20260924 03:09:04.766211 2643 ts_tablet_manager.cc:1428] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303: Time spent starting tablet: real 0.015s user 0.013s sys 0.004s
I20260924 03:09:04.769500 2644 ts_tablet_manager.cc:1428] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479: Time spent starting tablet: real 0.018s user 0.013s sys 0.000s
I20260924 03:09:04.773936 2647 raft_consensus.cc:348] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } }
I20260924 03:09:04.774303 2647 raft_consensus.cc:374] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:04.774447 2647 raft_consensus.cc:729] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c77b6c7f2fd44fcb8e81719b50e9b88, State: Initialized, Role: FOLLOWER
I20260924 03:09:04.775043 2647 consensus_queue.cc:260] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } }
I20260924 03:09:04.776755 2647 ts_tablet_manager.cc:1428] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
W20260924 03:09:04.790705 2347 tablet.cc:2369] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:09:04.829038 2486 tablet.cc:2369] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:09:04.870975 2625 tablet.cc:2369] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:09:04.915220 2649 raft_consensus.cc:482] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:04.915606 2649 raft_consensus.cc:504] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } }
I20260924 03:09:04.917376 2649 leader_election.cc:288] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff49ff7c1f3c46deb3d651443b6f8303 (127.0.97.3:40409), 1c77b6c7f2fd44fcb8e81719b50e9b88 (127.0.97.1:42511)
I20260924 03:09:04.924786 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.917975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55575 (local address 127.0.97.3:40409)
0924 03:09:04.918179 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:04.918184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:04.918770 (+ 586us) server_negotiation.cc:407] Connection header received
0924 03:09:04.919158 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:04.919164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:04.919243 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:04.919317 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:04.919914 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.921004 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.922990 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.923767 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.923921 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.924035 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:04.924322 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 03:09:04.924440 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 03:09:04.924805 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.919086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40813 (local address 127.0.97.1:42511)
0924 03:09:04.919293 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:04.919299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:04.919518 (+ 219us) server_negotiation.cc:407] Connection header received
0924 03:09:04.919761 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:04.919765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:04.919830 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:04.919952 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:04.920357 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.921318 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.923096 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.923839 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.924007 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.924112 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:04.924387 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 03:09:04.924541 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:09:04.924917 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.917865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40409 (local address 127.0.97.2:55575)
0924 03:09:04.918634 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:04.918655 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:04.918929 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:04.919473 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:04.919476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:04.919495 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:04.919775 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.919782 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.921138 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.921141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.922745 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.922756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.923987 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.923993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.924156 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.924179 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 03:09:04.924290 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 03:09:04.924420 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":57,"thread_start_us":421,"threads_started":1}
I20260924 03:09:04.925384 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.918771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42511 (local address 127.0.97.2:40813)
0924 03:09:04.919390 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:04.919403 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:04.919608 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:04.919956 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:04.919958 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:04.919974 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:04.920224 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.920230 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.921462 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.921468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.922948 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.922959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.924074 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.924080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.924236 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.924259 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 03:09:04.924364 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 03:09:04.924517 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":21,"thread_start_us":301,"threads_started":1}
I20260924 03:09:04.926005 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18041d4cf18547b7b52daa36ea54cda5" candidate_uuid: "b3b67e6f90164055b29ca57d13548479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" is_pre_election: true
I20260924 03:09:04.926337 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18041d4cf18547b7b52daa36ea54cda5" candidate_uuid: "b3b67e6f90164055b29ca57d13548479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" is_pre_election: true
I20260924 03:09:04.926625 2579 raft_consensus.cc:2454] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b67e6f90164055b29ca57d13548479 in term 0.
I20260924 03:09:04.926863 2301 raft_consensus.cc:2454] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b67e6f90164055b29ca57d13548479 in term 0.
I20260924 03:09:04.927381 2367 leader_election.cc:302] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3b67e6f90164055b29ca57d13548479, ff49ff7c1f3c46deb3d651443b6f8303; no voters:
I20260924 03:09:04.927752 2649 raft_consensus.cc:2781] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:04.927942 2649 raft_consensus.cc:482] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:04.928043 2649 raft_consensus.cc:3037] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:04.930521 2649 raft_consensus.cc:504] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } }
I20260924 03:09:04.931162 2649 leader_election.cc:288] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 [CANDIDATE]: Term 1 election: Requested vote from peers ff49ff7c1f3c46deb3d651443b6f8303 (127.0.97.3:40409), 1c77b6c7f2fd44fcb8e81719b50e9b88 (127.0.97.1:42511)
I20260924 03:09:04.931710 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18041d4cf18547b7b52daa36ea54cda5" candidate_uuid: "b3b67e6f90164055b29ca57d13548479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff49ff7c1f3c46deb3d651443b6f8303"
I20260924 03:09:04.931922 2579 raft_consensus.cc:3037] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:04.932076 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18041d4cf18547b7b52daa36ea54cda5" candidate_uuid: "b3b67e6f90164055b29ca57d13548479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88"
I20260924 03:09:04.932291 2301 raft_consensus.cc:3037] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:04.935252 2579 raft_consensus.cc:2454] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b67e6f90164055b29ca57d13548479 in term 1.
I20260924 03:09:04.935518 2301 raft_consensus.cc:2454] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b67e6f90164055b29ca57d13548479 in term 1.
I20260924 03:09:04.935870 2367 leader_election.cc:302] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3b67e6f90164055b29ca57d13548479, ff49ff7c1f3c46deb3d651443b6f8303; no voters:
I20260924 03:09:04.936201 2649 raft_consensus.cc:2781] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:04.936759 2649 raft_consensus.cc:686] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 [term 1 LEADER]: Becoming Leader. State: Replica: b3b67e6f90164055b29ca57d13548479, State: Running, Role: LEADER
I20260924 03:09:04.937235 2649 consensus_queue.cc:237] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } }
I20260924 03:09:04.941954 2139 catalog_manager.cc:5495] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b67e6f90164055b29ca57d13548479 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b3b67e6f90164055b29ca57d13548479" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } health_report { overall_health: UNKNOWN } } }
Sep 24 03:09:04 dist-test-slave-5l98 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:04.989693 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.974922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37099 (local address 127.0.0.1:58678)
0924 03:09:04.975604 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:04.975620 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:04.975742 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:04.976147 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:04.976150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:04.976983 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:04.977249 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.977258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.978388 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.978391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.979326 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.979336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.980150 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.980155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.980265 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.981009 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:04.981026 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:04.983318 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:04.987195 (+ 3877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:04.987206 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:04.987213 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:04.987630 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:04.988120 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:04.988124 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:04.988126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:04.988272 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:04.988842 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:04.988849 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:04.988994 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 03:09:04.989133 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 03:09:04.989358 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":405,"threads_started":1}
I20260924 03:09:04.989825 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.974933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58678 (local address 127.0.97.62:37099)
0924 03:09:04.975143 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:04.975151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:04.975704 (+ 553us) server_negotiation.cc:407] Connection header received
0924 03:09:04.975910 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:04.975914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:04.975992 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:04.976084 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:04.977420 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.978257 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.979466 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.979981 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.980083 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.983514 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:04.983518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:04.983522 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:04.983550 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:04.987060 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:04.987742 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:04.987746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:04.987748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:04.987925 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:04.988381 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:04.988384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:04.988386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:04.988710 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:04.988817 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:04.989229 (+ 412us) server_negotiation.cc:299] Negotiation successful
0924 03:09:04.989479 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":73}
I20260924 03:09:04.995680 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58678:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 03:09:04.996826 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:05.003996 2276 tablet_service.cc:1467] Processing CreateTablet for tablet c0b4114507264291a4b7eb1998b75aef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2a8d2431663e479b9b9d3f86223d0bb7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:09:05.004251 2415 tablet_service.cc:1467] Processing CreateTablet for tablet c0b4114507264291a4b7eb1998b75aef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2a8d2431663e479b9b9d3f86223d0bb7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:09:05.005095 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b4114507264291a4b7eb1998b75aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:05.005091 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b4114507264291a4b7eb1998b75aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:05.004752 2554 tablet_service.cc:1467] Processing CreateTablet for tablet c0b4114507264291a4b7eb1998b75aef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2a8d2431663e479b9b9d3f86223d0bb7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:09:05.005688 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b4114507264291a4b7eb1998b75aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:05.011638 2643 tablet_bootstrap.cc:492] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303: Bootstrap starting.
I20260924 03:09:05.012523 2647 tablet_bootstrap.cc:492] T c0b4114507264291a4b7eb1998b75aef P 1c77b6c7f2fd44fcb8e81719b50e9b88: Bootstrap starting.
I20260924 03:09:05.012523 2644 tablet_bootstrap.cc:492] T c0b4114507264291a4b7eb1998b75aef P b3b67e6f90164055b29ca57d13548479: Bootstrap starting.
I20260924 03:09:05.013235 2643 tablet_bootstrap.cc:654] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:05.014192 2644 tablet_bootstrap.cc:654] T c0b4114507264291a4b7eb1998b75aef P b3b67e6f90164055b29ca57d13548479: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:05.014192 2647 tablet_bootstrap.cc:654] T c0b4114507264291a4b7eb1998b75aef P 1c77b6c7f2fd44fcb8e81719b50e9b88: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:05.016978 2647 tablet_bootstrap.cc:492] T c0b4114507264291a4b7eb1998b75aef P 1c77b6c7f2fd44fcb8e81719b50e9b88: No bootstrap required, opened a new log
I20260924 03:09:05.017143 2647 ts_tablet_manager.cc:1397] T c0b4114507264291a4b7eb1998b75aef P 1c77b6c7f2fd44fcb8e81719b50e9b88: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 03:09:05.017266 2643 tablet_bootstrap.cc:492] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303: No bootstrap required, opened a new log
I20260924 03:09:05.017431 2643 ts_tablet_manager.cc:1397] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 03:09:05.017582 2644 tablet_bootstrap.cc:492] T c0b4114507264291a4b7eb1998b75aef P b3b67e6f90164055b29ca57d13548479: No bootstrap required, opened a new log
I20260924 03:09:05.017720 2644 ts_tablet_manager.cc:1397] T c0b4114507264291a4b7eb1998b75aef P b3b67e6f90164055b29ca57d13548479: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260924 03:09:05.017984 2647 raft_consensus.cc:348] T c0b4114507264291a4b7eb1998b75aef P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:05.018218 2647 raft_consensus.cc:374] T c0b4114507264291a4b7eb1998b75aef P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:05.018316 2647 raft_consensus.cc:729] T c0b4114507264291a4b7eb1998b75aef P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c77b6c7f2fd44fcb8e81719b50e9b88, State: Initialized, Role: FOLLOWER
I20260924 03:09:05.018189 2643 raft_consensus.cc:348] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:05.018430 2643 raft_consensus.cc:374] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:05.018524 2643 raft_consensus.cc:729] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff49ff7c1f3c46deb3d651443b6f8303, State: Initialized, Role: FOLLOWER
I20260924 03:09:05.018536 2647 consensus_queue.cc:260] T c0b4114507264291a4b7eb1998b75aef P 1c77b6c7f2fd44fcb8e81719b50e9b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:05.018772 2643 consensus_queue.cc:260] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:05.019167 2647 ts_tablet_manager.cc:1428] T c0b4114507264291a4b7eb1998b75aef P 1c77b6c7f2fd44fcb8e81719b50e9b88: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260924 03:09:05.019266 2650 tablet_replica.cc:439] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } } }
I20260924 03:09:05.019647 2650 tablet_replica.cc:442] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:05.019507 2644 raft_consensus.cc:348] T c0b4114507264291a4b7eb1998b75aef P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:05.019734 2644 raft_consensus.cc:374] T c0b4114507264291a4b7eb1998b75aef P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:05.019826 2644 raft_consensus.cc:729] T c0b4114507264291a4b7eb1998b75aef P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b67e6f90164055b29ca57d13548479, State: Initialized, Role: FOLLOWER
I20260924 03:09:05.020107 2643 ts_tablet_manager.cc:1428] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 03:09:05.020108 2651 tablet_replica.cc:439] T c0b4114507264291a4b7eb1998b75aef P 1c77b6c7f2fd44fcb8e81719b50e9b88: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } } }
I20260924 03:09:05.020332 2644 consensus_queue.cc:260] T c0b4114507264291a4b7eb1998b75aef P b3b67e6f90164055b29ca57d13548479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:05.020514 2651 tablet_replica.cc:442] T c0b4114507264291a4b7eb1998b75aef P 1c77b6c7f2fd44fcb8e81719b50e9b88: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:05.020715 2649 tablet_replica.cc:439] T c0b4114507264291a4b7eb1998b75aef P b3b67e6f90164055b29ca57d13548479: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } } }
I20260924 03:09:05.020924 2644 ts_tablet_manager.cc:1428] T c0b4114507264291a4b7eb1998b75aef P b3b67e6f90164055b29ca57d13548479: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 03:09:05.020993 2649 tablet_replica.cc:442] T c0b4114507264291a4b7eb1998b75aef P b3b67e6f90164055b29ca57d13548479: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:05.036399 2650 raft_consensus.cc:482] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:05.036724 2650 raft_consensus.cc:504] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:05.038318 2650 leader_election.cc:288] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c77b6c7f2fd44fcb8e81719b50e9b88 (127.0.97.1:42511), b3b67e6f90164055b29ca57d13548479 (127.0.97.2:44791)
I20260924 03:09:05.045300 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.038762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43531 (local address 127.0.97.1:42511)
0924 03:09:05.039026 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.039031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.039052 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:09:05.039125 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.039130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.039194 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.039319 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:05.039823 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.040865 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.043345 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.044239 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.044414 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.044553 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:05.044862 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 03:09:05.044997 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260924 03:09:05.045395 2513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.038655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42511 (local address 127.0.97.3:43531)
0924 03:09:05.038838 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:05.038859 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:05.039005 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:05.039383 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:05.039387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:05.039414 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:05.039661 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.039678 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.041009 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.041013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.043157 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.043170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.044414 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.044421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.044674 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.044702 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 03:09:05.044826 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 03:09:05.044978 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":3}
I20260924 03:09:05.045511 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.039282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54893 (local address 127.0.97.2:44791)
0924 03:09:05.039539 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.039548 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.039957 (+ 409us) server_negotiation.cc:407] Connection header received
0924 03:09:05.040059 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.040064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.040139 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.040232 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:05.040857 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.041815 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.043747 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.044563 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.044721 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.044836 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:05.045126 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 03:09:05.045244 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 03:09:05.046252 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.039059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44791 (local address 127.0.97.3:54893)
0924 03:09:05.039854 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:05.039866 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:05.039952 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:05.040295 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:05.040301 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:05.040329 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:05.040699 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.040710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.041978 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.041984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.043586 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.043598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.044806 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.044813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.044981 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.045003 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 03:09:05.045104 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 03:09:05.045232 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":535,"threads_started":1}
I20260924 03:09:05.046155 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b4114507264291a4b7eb1998b75aef" candidate_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" is_pre_election: true
I20260924 03:09:05.046460 2301 raft_consensus.cc:2454] T c0b4114507264291a4b7eb1998b75aef P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff49ff7c1f3c46deb3d651443b6f8303 in term 0.
I20260924 03:09:05.046972 2506 leader_election.cc:302] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c77b6c7f2fd44fcb8e81719b50e9b88, ff49ff7c1f3c46deb3d651443b6f8303; no voters:
I20260924 03:09:05.047350 2650 raft_consensus.cc:2781] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:05.047219 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b4114507264291a4b7eb1998b75aef" candidate_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b67e6f90164055b29ca57d13548479" is_pre_election: true
I20260924 03:09:05.047463 2650 raft_consensus.cc:482] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:05.047570 2650 raft_consensus.cc:3037] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:05.047701 2439 raft_consensus.cc:2454] T c0b4114507264291a4b7eb1998b75aef P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff49ff7c1f3c46deb3d651443b6f8303 in term 0.
I20260924 03:09:05.050241 2650 raft_consensus.cc:504] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:05.050843 2650 leader_election.cc:288] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303 [CANDIDATE]: Term 1 election: Requested vote from peers 1c77b6c7f2fd44fcb8e81719b50e9b88 (127.0.97.1:42511), b3b67e6f90164055b29ca57d13548479 (127.0.97.2:44791)
I20260924 03:09:05.051314 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b4114507264291a4b7eb1998b75aef" candidate_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b67e6f90164055b29ca57d13548479"
I20260924 03:09:05.051337 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b4114507264291a4b7eb1998b75aef" candidate_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88"
I20260924 03:09:05.051530 2439 raft_consensus.cc:3037] T c0b4114507264291a4b7eb1998b75aef P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:05.051604 2301 raft_consensus.cc:3037] T c0b4114507264291a4b7eb1998b75aef P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:05.053743 2439 raft_consensus.cc:2454] T c0b4114507264291a4b7eb1998b75aef P b3b67e6f90164055b29ca57d13548479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff49ff7c1f3c46deb3d651443b6f8303 in term 1.
I20260924 03:09:05.053987 2301 raft_consensus.cc:2454] T c0b4114507264291a4b7eb1998b75aef P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff49ff7c1f3c46deb3d651443b6f8303 in term 1.
I20260924 03:09:05.054144 2506 leader_election.cc:302] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3b67e6f90164055b29ca57d13548479, ff49ff7c1f3c46deb3d651443b6f8303; no voters:
I20260924 03:09:05.054425 2650 raft_consensus.cc:2781] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:05.054853 2650 raft_consensus.cc:686] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303 [term 1 LEADER]: Becoming Leader. State: Replica: ff49ff7c1f3c46deb3d651443b6f8303, State: Running, Role: LEADER
I20260924 03:09:05.055348 2650 consensus_queue.cc:237] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:05.058321 2666 tablet_replica.cc:439] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303: TxnStatusTablet state changed. Reason: New leader ff49ff7c1f3c46deb3d651443b6f8303. Latest consensus state: current_term: 1 leader_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } } }
I20260924 03:09:05.058671 2666 tablet_replica.cc:442] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:09:05.059495 2668 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 03:09:05.059751 2668 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 03:09:05.060261 2137 catalog_manager.cc:5495] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff49ff7c1f3c46deb3d651443b6f8303 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } health_report { overall_health: HEALTHY } } }
I20260924 03:09:05.090039 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.083497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59708 (local address 127.0.97.3:40409)
0924 03:09:05.083689 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.083697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.083993 (+ 296us) server_negotiation.cc:407] Connection header received
0924 03:09:05.084321 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.084327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.084448 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.084610 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:05.084994 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.086131 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.087678 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.088180 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.088314 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.088845 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:05.089376 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:05.089548 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:05.089627 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 03:09:05.089710 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 03:09:05.090082 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.083505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40409 (local address 127.0.0.1:59708)
0924 03:09:05.083898 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:05.083915 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:05.084103 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:05.084629 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:05.084633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:05.084645 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:09:05.084824 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.084833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.086293 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.086298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.087540 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.087552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.088327 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.088332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.088562 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.088635 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:09:05.089501 (+ 866us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:09:05.089523 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 03:09:05.089609 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 03:09:05.089759 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215}
I20260924 03:09:05.098047 2301 raft_consensus.cc:1261] T c0b4114507264291a4b7eb1998b75aef P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 1 FOLLOWER]: Refusing update from remote peer ff49ff7c1f3c46deb3d651443b6f8303: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:05.098002 2439 raft_consensus.cc:1261] T c0b4114507264291a4b7eb1998b75aef P b3b67e6f90164055b29ca57d13548479 [term 1 FOLLOWER]: Refusing update from remote peer ff49ff7c1f3c46deb3d651443b6f8303: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:05.099080 2666 consensus_queue.cc:1035] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:05.099416 2650 consensus_queue.cc:1035] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:05.107827 2651 tablet_replica.cc:439] T c0b4114507264291a4b7eb1998b75aef P 1c77b6c7f2fd44fcb8e81719b50e9b88: TxnStatusTablet state changed. Reason: New leader ff49ff7c1f3c46deb3d651443b6f8303. Latest consensus state: current_term: 1 leader_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } } }
I20260924 03:09:05.108134 2651 tablet_replica.cc:442] T c0b4114507264291a4b7eb1998b75aef P 1c77b6c7f2fd44fcb8e81719b50e9b88: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:05.112372 2666 tablet_replica.cc:439] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } } }
I20260924 03:09:05.112686 2666 tablet_replica.cc:442] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:09:05.118901 2651 tablet_replica.cc:439] T c0b4114507264291a4b7eb1998b75aef P 1c77b6c7f2fd44fcb8e81719b50e9b88: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } } }
I20260924 03:09:05.119127 2651 tablet_replica.cc:442] T c0b4114507264291a4b7eb1998b75aef P 1c77b6c7f2fd44fcb8e81719b50e9b88: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:05.119760 2649 tablet_replica.cc:439] T c0b4114507264291a4b7eb1998b75aef P b3b67e6f90164055b29ca57d13548479: TxnStatusTablet state changed. Reason: New leader ff49ff7c1f3c46deb3d651443b6f8303. Latest consensus state: current_term: 1 leader_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } } }
I20260924 03:09:05.120065 2649 tablet_replica.cc:442] T c0b4114507264291a4b7eb1998b75aef P b3b67e6f90164055b29ca57d13548479: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:05.121958 2671 mvcc.cc:205] Tried to move back new op lower bound from 7332738437511536640 to 7332738437350596608. Current Snapshot: MvccSnapshot[applied={T|T < 7332738437511536640 or (T in {7332738437511536640})}]
I20260924 03:09:05.123397 2650 tablet_replica.cc:439] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } } }
I20260924 03:09:05.123637 2650 tablet_replica.cc:442] T c0b4114507264291a4b7eb1998b75aef P ff49ff7c1f3c46deb3d651443b6f8303: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:09:05.125984 2673 mvcc.cc:205] Tried to move back new op lower bound from 7332738437511536640 to 7332738437350596608. Current Snapshot: MvccSnapshot[applied={T|T < 7332738437511536640}]
I20260924 03:09:05.127815 2649 tablet_replica.cc:439] T c0b4114507264291a4b7eb1998b75aef P b3b67e6f90164055b29ca57d13548479: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } } }
I20260924 03:09:05.128073 2649 tablet_replica.cc:442] T c0b4114507264291a4b7eb1998b75aef P b3b67e6f90164055b29ca57d13548479: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:05.131613 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.124207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55282 (local address 127.0.97.2:44791)
0924 03:09:05.124370 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.124376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.124394 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:09:05.124516 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.124522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.124638 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.124787 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:05.125765 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.126952 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.128926 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.129578 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.129701 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.130298 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:05.131015 (+ 717us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:05.131163 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:05.131248 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 03:09:05.131323 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 03:09:05.148623 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.142280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40409 (local address 127.0.97.2:33833)
0924 03:09:05.143026 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:05.143042 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:05.143204 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:05.143560 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:05.143563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:05.143572 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:09:05.143845 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.143852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.145356 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.145360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.146556 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.146567 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.147525 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.147529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.147635 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.147687 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:09:05.148082 (+ 395us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:09:05.148089 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 03:09:05.148146 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 03:09:05.148252 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":378,"threads_started":1}
I20260924 03:09:05.148663 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.142314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33833 (local address 127.0.97.3:40409)
0924 03:09:05.142498 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.142504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.143119 (+ 615us) server_negotiation.cc:407] Connection header received
0924 03:09:05.143332 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.143338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.143445 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.143541 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:05.144070 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.145212 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.146683 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.147358 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.147484 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.147867 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:05.147989 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:05.148075 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:05.148197 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 03:09:05.148277 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:09:05.158229 2301 raft_consensus.cc:1261] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 1 FOLLOWER]: Refusing update from remote peer b3b67e6f90164055b29ca57d13548479: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:05.158804 2579 raft_consensus.cc:1261] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303 [term 1 FOLLOWER]: Refusing update from remote peer b3b67e6f90164055b29ca57d13548479: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:05.158844 2649 consensus_queue.cc:1035] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:05.159498 2649 consensus_queue.cc:1035] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:05.203691 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.194367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54303 (local address 127.0.97.2:44791)
0924 03:09:05.194538 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.194552 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.197319 (+ 2767us) server_negotiation.cc:407] Connection header received
0924 03:09:05.197536 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.197541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.197638 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.197733 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:05.198377 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.199454 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.201071 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.201710 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.201840 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.202961 (+ 1121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:05.203095 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:05.203194 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:05.203408 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 03:09:05.203475 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 03:09:05.204001 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.194482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44791 (local address 127.0.97.3:54303)
0924 03:09:05.197176 (+ 2694us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:05.197217 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:05.197385 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:05.197771 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:05.197776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:05.197791 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:09:05.198205 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.198217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.199587 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.199592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.200928 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.200941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.202081 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.202092 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.202612 (+ 520us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.202687 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:09:05.203256 (+ 569us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:09:05.203274 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 03:09:05.203355 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 03:09:05.203510 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2494,"thread_start_us":2207,"threads_started":1}
I20260924 03:09:05.247200 2139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:05.247473 2139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:05.251412 2139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9: Sending DeleteTablet for 3 replicas of tablet 18041d4cf18547b7b52daa36ea54cda5
I20260924 03:09:05.252731 2415 tablet_service.cc:1514] Processing DeleteTablet for tablet 18041d4cf18547b7b52daa36ea54cda5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55266
I20260924 03:09:05.253032 2276 tablet_service.cc:1514] Processing DeleteTablet for tablet 18041d4cf18547b7b52daa36ea54cda5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35090
I20260924 03:09:05.255208 2554 tablet_service.cc:1514] Processing DeleteTablet for tablet 18041d4cf18547b7b52daa36ea54cda5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59700
I20260924 03:09:05.256120 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:09:05.257292 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:05.268396 2692 tablet_replica.cc:330] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88: stopping tablet replica
I20260924 03:09:05.268924 2692 raft_consensus.cc:2229] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:05.269279 2692 raft_consensus.cc:2258] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:05.270426 2414 tablet_service.cc:1467] Processing CreateTablet for tablet 342d983fea894d9f84f50cdc4dc280d6 (DEFAULT_TABLE table=test-table [id=bd4bf1d685254daba2da7aea82263703]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:05.270977 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 342d983fea894d9f84f50cdc4dc280d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:05.271123 2553 tablet_service.cc:1467] Processing CreateTablet for tablet 342d983fea894d9f84f50cdc4dc280d6 (DEFAULT_TABLE table=test-table [id=bd4bf1d685254daba2da7aea82263703]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:05.271667 2553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 342d983fea894d9f84f50cdc4dc280d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:05.272670 2276 tablet_service.cc:1467] Processing CreateTablet for tablet 342d983fea894d9f84f50cdc4dc280d6 (DEFAULT_TABLE table=test-table [id=bd4bf1d685254daba2da7aea82263703]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:05.273488 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 342d983fea894d9f84f50cdc4dc280d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:05.278908 2643 tablet_bootstrap.cc:492] T 342d983fea894d9f84f50cdc4dc280d6 P ff49ff7c1f3c46deb3d651443b6f8303: Bootstrap starting.
I20260924 03:09:05.280387 2644 tablet_bootstrap.cc:492] T 342d983fea894d9f84f50cdc4dc280d6 P b3b67e6f90164055b29ca57d13548479: Bootstrap starting.
I20260924 03:09:05.280683 2643 tablet_bootstrap.cc:654] T 342d983fea894d9f84f50cdc4dc280d6 P ff49ff7c1f3c46deb3d651443b6f8303: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:05.282501 2644 tablet_bootstrap.cc:654] T 342d983fea894d9f84f50cdc4dc280d6 P b3b67e6f90164055b29ca57d13548479: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:05.290017 2693 tablet_replica.cc:330] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303: stopping tablet replica
I20260924 03:09:05.290503 2647 tablet_bootstrap.cc:492] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Bootstrap starting.
I20260924 03:09:05.290903 2692 ts_tablet_manager.cc:1905] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:05.292678 2647 tablet_bootstrap.cc:654] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:05.297521 2691 tablet_replica.cc:330] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479: stopping tablet replica
I20260924 03:09:05.297966 2691 raft_consensus.cc:2229] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:09:05.298635 2691 raft_consensus.cc:2258] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:05.299472 2644 tablet_bootstrap.cc:492] T 342d983fea894d9f84f50cdc4dc280d6 P b3b67e6f90164055b29ca57d13548479: No bootstrap required, opened a new log
I20260924 03:09:05.299626 2644 ts_tablet_manager.cc:1397] T 342d983fea894d9f84f50cdc4dc280d6 P b3b67e6f90164055b29ca57d13548479: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.000s
I20260924 03:09:05.301497 2644 raft_consensus.cc:348] T 342d983fea894d9f84f50cdc4dc280d6 P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:05.301744 2644 raft_consensus.cc:374] T 342d983fea894d9f84f50cdc4dc280d6 P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:05.301831 2644 raft_consensus.cc:729] T 342d983fea894d9f84f50cdc4dc280d6 P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b67e6f90164055b29ca57d13548479, State: Initialized, Role: FOLLOWER
I20260924 03:09:05.302160 2644 consensus_queue.cc:260] T 342d983fea894d9f84f50cdc4dc280d6 P b3b67e6f90164055b29ca57d13548479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:05.303191 2647 tablet_bootstrap.cc:492] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88: No bootstrap required, opened a new log
I20260924 03:09:05.303407 2647 ts_tablet_manager.cc:1397] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.006s
I20260924 03:09:05.304260 2647 raft_consensus.cc:348] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:05.304617 2647 raft_consensus.cc:374] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:05.304754 2647 raft_consensus.cc:729] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c77b6c7f2fd44fcb8e81719b50e9b88, State: Initialized, Role: FOLLOWER
I20260924 03:09:05.305163 2647 consensus_queue.cc:260] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:05.305424 2644 ts_tablet_manager.cc:1428] T 342d983fea894d9f84f50cdc4dc280d6 P b3b67e6f90164055b29ca57d13548479: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260924 03:09:05.308171 2647 ts_tablet_manager.cc:1428] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260924 03:09:05.301210 2643 tablet_bootstrap.cc:492] T 342d983fea894d9f84f50cdc4dc280d6 P ff49ff7c1f3c46deb3d651443b6f8303: No bootstrap required, opened a new log
I20260924 03:09:05.309414 2691 ts_tablet_manager.cc:1905] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:05.309597 2643 ts_tablet_manager.cc:1397] T 342d983fea894d9f84f50cdc4dc280d6 P ff49ff7c1f3c46deb3d651443b6f8303: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.005s
I20260924 03:09:05.310657 2643 raft_consensus.cc:348] T 342d983fea894d9f84f50cdc4dc280d6 P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:05.310889 2643 raft_consensus.cc:374] T 342d983fea894d9f84f50cdc4dc280d6 P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:05.310976 2643 raft_consensus.cc:729] T 342d983fea894d9f84f50cdc4dc280d6 P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff49ff7c1f3c46deb3d651443b6f8303, State: Initialized, Role: FOLLOWER
I20260924 03:09:05.311229 2643 consensus_queue.cc:260] T 342d983fea894d9f84f50cdc4dc280d6 P ff49ff7c1f3c46deb3d651443b6f8303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:05.311812 2643 ts_tablet_manager.cc:1428] T 342d983fea894d9f84f50cdc4dc280d6 P ff49ff7c1f3c46deb3d651443b6f8303: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 03:09:05.312731 2693 raft_consensus.cc:2229] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:05.313051 2693 raft_consensus.cc:2258] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:05.315635 2693 ts_tablet_manager.cc:1905] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:05.321710 2692 ts_tablet_manager.cc:1918] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:09:05.321918 2692 log.cc:1196] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/18041d4cf18547b7b52daa36ea54cda5
I20260924 03:09:05.322007 2693 ts_tablet_manager.cc:1918] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:09:05.322175 2693 log.cc:1196] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/18041d4cf18547b7b52daa36ea54cda5
I20260924 03:09:05.322561 2692 ts_tablet_manager.cc:1939] T 18041d4cf18547b7b52daa36ea54cda5 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Deleting consensus metadata
I20260924 03:09:05.322726 2693 ts_tablet_manager.cc:1939] T 18041d4cf18547b7b52daa36ea54cda5 P ff49ff7c1f3c46deb3d651443b6f8303: Deleting consensus metadata
I20260924 03:09:05.324460 2124 catalog_manager.cc:4841] TS 1c77b6c7f2fd44fcb8e81719b50e9b88 (127.0.97.1:42511): tablet 18041d4cf18547b7b52daa36ea54cda5 (table test-table [id=d5b3fd7228104af5bd5d786563313378]) successfully deleted
I20260924 03:09:05.324800 2124 catalog_manager.cc:4841] TS ff49ff7c1f3c46deb3d651443b6f8303 (127.0.97.3:40409): tablet 18041d4cf18547b7b52daa36ea54cda5 (table test-table [id=d5b3fd7228104af5bd5d786563313378]) successfully deleted
I20260924 03:09:05.327231 2691 ts_tablet_manager.cc:1918] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:09:05.327421 2691 log.cc:1196] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/18041d4cf18547b7b52daa36ea54cda5
I20260924 03:09:05.327888 2691 ts_tablet_manager.cc:1939] T 18041d4cf18547b7b52daa36ea54cda5 P b3b67e6f90164055b29ca57d13548479: Deleting consensus metadata
I20260924 03:09:05.329406 2124 catalog_manager.cc:4841] TS b3b67e6f90164055b29ca57d13548479 (127.0.97.2:44791): tablet 18041d4cf18547b7b52daa36ea54cda5 (table test-table [id=d5b3fd7228104af5bd5d786563313378]) successfully deleted
I20260924 03:09:05.412876 2651 raft_consensus.cc:482] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:05.413220 2651 raft_consensus.cc:504] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:05.415271 2651 leader_election.cc:288] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3b67e6f90164055b29ca57d13548479 (127.0.97.2:44791), ff49ff7c1f3c46deb3d651443b6f8303 (127.0.97.3:40409)
I20260924 03:09:05.425901 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.415698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40367 (local address 127.0.97.2:44791)
0924 03:09:05.415932 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.415939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.416689 (+ 750us) server_negotiation.cc:407] Connection header received
0924 03:09:05.417015 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.417023 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.417111 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.417211 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:05.418134 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.419515 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.422689 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.424369 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.424528 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.424653 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:05.425393 (+ 740us) server_negotiation.cc:299] Negotiation successful
0924 03:09:05.425558 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 03:09:05.427090 2697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.415654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44791 (local address 127.0.97.1:40367)
0924 03:09:05.416568 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:05.416591 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:05.416827 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:05.417446 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:05.417452 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:05.417480 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:05.417914 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.417926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.419759 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.419765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.422462 (+ 2697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.422483 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.424911 (+ 2428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.424918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.425143 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.425187 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 03:09:05.426463 (+ 1276us) client_negotiation.cc:240] Negotiation successful
0924 03:09:05.426663 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":525,"threads_started":1}
I20260924 03:09:05.428690 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342d983fea894d9f84f50cdc4dc280d6" candidate_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b67e6f90164055b29ca57d13548479" is_pre_election: true
I20260924 03:09:05.429014 2439 raft_consensus.cc:2454] T 342d983fea894d9f84f50cdc4dc280d6 P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c77b6c7f2fd44fcb8e81719b50e9b88 in term 0.
I20260924 03:09:05.429737 2227 leader_election.cc:302] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c77b6c7f2fd44fcb8e81719b50e9b88, b3b67e6f90164055b29ca57d13548479; no voters:
I20260924 03:09:05.430341 2651 raft_consensus.cc:2781] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:05.430470 2651 raft_consensus.cc:482] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:05.430591 2651 raft_consensus.cc:3037] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:05.434743 2651 raft_consensus.cc:504] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:05.435703 2651 leader_election.cc:288] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [CANDIDATE]: Term 1 election: Requested vote from peers b3b67e6f90164055b29ca57d13548479 (127.0.97.2:44791), ff49ff7c1f3c46deb3d651443b6f8303 (127.0.97.3:40409)
I20260924 03:09:05.436645 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342d983fea894d9f84f50cdc4dc280d6" candidate_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b67e6f90164055b29ca57d13548479"
I20260924 03:09:05.436833 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.417121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38409 (local address 127.0.97.3:40409)
0924 03:09:05.417313 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.417319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.427871 (+ 10552us) server_negotiation.cc:407] Connection header received
0924 03:09:05.428507 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.428516 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.428603 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.428670 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:05.429533 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.430830 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.433857 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.435395 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.435524 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.435659 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:05.436326 (+ 667us) server_negotiation.cc:299] Negotiation successful
0924 03:09:05.436521 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 03:09:05.436889 2439 raft_consensus.cc:3037] T 342d983fea894d9f84f50cdc4dc280d6 P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:05.437636 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.416701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40409 (local address 127.0.97.1:38409)
0924 03:09:05.427755 (+ 11054us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:05.427777 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:05.428300 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:05.428907 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:05.428912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:05.428943 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:05.429278 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.429289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.431057 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.431062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.433635 (+ 2573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.433655 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.435893 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.435901 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.436122 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.436163 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 03:09:05.437107 (+ 944us) client_negotiation.cc:240] Negotiation successful
0924 03:09:05.437265 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10893,"thread_start_us":11192,"threads_started":1}
I20260924 03:09:05.440516 2439 raft_consensus.cc:2454] T 342d983fea894d9f84f50cdc4dc280d6 P b3b67e6f90164055b29ca57d13548479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c77b6c7f2fd44fcb8e81719b50e9b88 in term 1.
I20260924 03:09:05.441279 2227 leader_election.cc:302] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c77b6c7f2fd44fcb8e81719b50e9b88, b3b67e6f90164055b29ca57d13548479; no voters:
I20260924 03:09:05.441632 2651 raft_consensus.cc:2781] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:05.442173 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342d983fea894d9f84f50cdc4dc280d6" candidate_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" is_pre_election: true
I20260924 03:09:05.442453 2579 raft_consensus.cc:2454] T 342d983fea894d9f84f50cdc4dc280d6 P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c77b6c7f2fd44fcb8e81719b50e9b88 in term 0.
I20260924 03:09:05.443192 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342d983fea894d9f84f50cdc4dc280d6" candidate_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff49ff7c1f3c46deb3d651443b6f8303"
I20260924 03:09:05.443406 2579 raft_consensus.cc:3037] T 342d983fea894d9f84f50cdc4dc280d6 P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:05.447194 2579 raft_consensus.cc:2454] T 342d983fea894d9f84f50cdc4dc280d6 P ff49ff7c1f3c46deb3d651443b6f8303 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c77b6c7f2fd44fcb8e81719b50e9b88 in term 1.
I20260924 03:09:05.448627 2651 raft_consensus.cc:686] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 1 LEADER]: Becoming Leader. State: Replica: 1c77b6c7f2fd44fcb8e81719b50e9b88, State: Running, Role: LEADER
I20260924 03:09:05.449172 2651 consensus_queue.cc:237] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:05.453459 2139 catalog_manager.cc:5495] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c77b6c7f2fd44fcb8e81719b50e9b88 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } health_report { overall_health: UNKNOWN } } }
I20260924 03:09:05.504612 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.496886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35106 (local address 127.0.97.1:42511)
0924 03:09:05.497448 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.497454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.497473 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:09:05.497585 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.497590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.497694 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.497871 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:05.498437 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.499564 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.501739 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.502388 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.502544 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.503095 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:05.503858 (+ 763us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:05.504017 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:05.504184 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 03:09:05.504274 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424}
I20260924 03:09:05.525350 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.517792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41429 (local address 127.0.97.3:40409)
0924 03:09:05.517984 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.517990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.518015 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:09:05.518287 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.518293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.518391 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.518473 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:05.519200 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.521138 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.523244 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.523816 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.523961 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.524547 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:05.524706 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:05.524798 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:05.524965 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 03:09:05.525045 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 03:09:05.525470 2703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.517430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40409 (local address 127.0.97.1:41429)
0924 03:09:05.517939 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:05.517960 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:05.518101 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:05.518686 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:05.518691 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:05.518704 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:09:05.519019 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.519034 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.521331 (+ 2297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.521338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.523074 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.523090 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.524030 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.524037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.524206 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.524311 (+ 105us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:09:05.524823 (+ 512us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:09:05.524835 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 03:09:05.524929 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 03:09:05.525072 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":603,"threads_started":1}
I20260924 03:09:05.537514 2439 raft_consensus.cc:1261] T 342d983fea894d9f84f50cdc4dc280d6 P b3b67e6f90164055b29ca57d13548479 [term 1 FOLLOWER]: Refusing update from remote peer 1c77b6c7f2fd44fcb8e81719b50e9b88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:05.537529 2579 raft_consensus.cc:1261] T 342d983fea894d9f84f50cdc4dc280d6 P ff49ff7c1f3c46deb3d651443b6f8303 [term 1 FOLLOWER]: Refusing update from remote peer 1c77b6c7f2fd44fcb8e81719b50e9b88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:05.538224 2651 consensus_queue.cc:1035] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:05.538614 2699 consensus_queue.cc:1035] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:05.554049 2672 mvcc.cc:205] Tried to move back new op lower bound from 7332738439314989056 to 7332738438964654080. Current Snapshot: MvccSnapshot[applied={T|T < 7332738439314989056}]
I20260924 03:09:05.591490 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.579406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56953 (local address 127.0.97.1:42511)
0924 03:09:05.579561 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.579567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.579586 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:09:05.579798 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.579804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.579887 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.579967 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:05.580527 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.581625 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.585154 (+ 3529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.585930 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.586045 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.590340 (+ 4295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:05.590502 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:05.590596 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:05.591074 (+ 478us) server_negotiation.cc:299] Negotiation successful
0924 03:09:05.591177 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 03:09:05.591540 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.579255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42511 (local address 127.0.97.3:56953)
0924 03:09:05.579489 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:05.579510 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:05.579631 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:05.580113 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:05.580119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:05.580133 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:09:05.580341 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.580353 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.583140 (+ 2787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.583145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.584815 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.584831 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.589874 (+ 5043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.589880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.590090 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.590143 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:09:05.590903 (+ 760us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:09:05.590918 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 03:09:05.591022 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 03:09:05.591181 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 03:09:05.593039 2672 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 03:09:07.249537 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:07.249779 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:07.254482 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9: Sending DeleteTablet for 3 replicas of tablet 342d983fea894d9f84f50cdc4dc280d6
I20260924 03:09:07.255627 2276 tablet_service.cc:1514] Processing DeleteTablet for tablet 342d983fea894d9f84f50cdc4dc280d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35090
I20260924 03:09:07.255713 2554 tablet_service.cc:1514] Processing DeleteTablet for tablet 342d983fea894d9f84f50cdc4dc280d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59700
I20260924 03:09:07.256130 2415 tablet_service.cc:1514] Processing DeleteTablet for tablet 342d983fea894d9f84f50cdc4dc280d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55266
I20260924 03:09:07.256331 2732 tablet_replica.cc:330] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88: stopping tablet replica
I20260924 03:09:07.256584 2732 raft_consensus.cc:2229] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:09:07.256641 2733 tablet_replica.cc:330] T 342d983fea894d9f84f50cdc4dc280d6 P ff49ff7c1f3c46deb3d651443b6f8303: stopping tablet replica
I20260924 03:09:07.256799 2733 raft_consensus.cc:2229] T 342d983fea894d9f84f50cdc4dc280d6 P ff49ff7c1f3c46deb3d651443b6f8303 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:07.257062 2733 raft_consensus.cc:2258] T 342d983fea894d9f84f50cdc4dc280d6 P ff49ff7c1f3c46deb3d651443b6f8303 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:07.257272 2732 raft_consensus.cc:2258] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:07.258114 2733 ts_tablet_manager.cc:1905] T 342d983fea894d9f84f50cdc4dc280d6 P ff49ff7c1f3c46deb3d651443b6f8303: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:07.258203 2732 ts_tablet_manager.cc:1905] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:07.265422 2732 ts_tablet_manager.cc:1918] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:09:07.265476 2733 ts_tablet_manager.cc:1918] T 342d983fea894d9f84f50cdc4dc280d6 P ff49ff7c1f3c46deb3d651443b6f8303: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:09:07.265676 2733 log.cc:1196] T 342d983fea894d9f84f50cdc4dc280d6 P ff49ff7c1f3c46deb3d651443b6f8303: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/342d983fea894d9f84f50cdc4dc280d6
I20260924 03:09:07.265755 2732 log.cc:1196] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/342d983fea894d9f84f50cdc4dc280d6
I20260924 03:09:07.266222 2732 ts_tablet_manager.cc:1939] T 342d983fea894d9f84f50cdc4dc280d6 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Deleting consensus metadata
I20260924 03:09:07.266222 2733 ts_tablet_manager.cc:1939] T 342d983fea894d9f84f50cdc4dc280d6 P ff49ff7c1f3c46deb3d651443b6f8303: Deleting consensus metadata
I20260924 03:09:07.267479 2124 catalog_manager.cc:4841] TS 1c77b6c7f2fd44fcb8e81719b50e9b88 (127.0.97.1:42511): tablet 342d983fea894d9f84f50cdc4dc280d6 (table test-table [id=bd4bf1d685254daba2da7aea82263703]) successfully deleted
I20260924 03:09:07.267778 2124 catalog_manager.cc:4841] TS ff49ff7c1f3c46deb3d651443b6f8303 (127.0.97.3:40409): tablet 342d983fea894d9f84f50cdc4dc280d6 (table test-table [id=bd4bf1d685254daba2da7aea82263703]) successfully deleted
I20260924 03:09:07.268376 2734 tablet_replica.cc:330] T 342d983fea894d9f84f50cdc4dc280d6 P b3b67e6f90164055b29ca57d13548479: stopping tablet replica
I20260924 03:09:07.268637 2734 raft_consensus.cc:2229] T 342d983fea894d9f84f50cdc4dc280d6 P b3b67e6f90164055b29ca57d13548479 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:07.268893 2734 raft_consensus.cc:2258] T 342d983fea894d9f84f50cdc4dc280d6 P b3b67e6f90164055b29ca57d13548479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:07.269778 2734 ts_tablet_manager.cc:1905] T 342d983fea894d9f84f50cdc4dc280d6 P b3b67e6f90164055b29ca57d13548479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:07.274724 2734 ts_tablet_manager.cc:1918] T 342d983fea894d9f84f50cdc4dc280d6 P b3b67e6f90164055b29ca57d13548479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:09:07.274947 2734 log.cc:1196] T 342d983fea894d9f84f50cdc4dc280d6 P b3b67e6f90164055b29ca57d13548479: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/342d983fea894d9f84f50cdc4dc280d6
I20260924 03:09:07.275396 2734 ts_tablet_manager.cc:1939] T 342d983fea894d9f84f50cdc4dc280d6 P b3b67e6f90164055b29ca57d13548479: Deleting consensus metadata
I20260924 03:09:07.276304 2124 catalog_manager.cc:4841] TS b3b67e6f90164055b29ca57d13548479 (127.0.97.2:44791): tablet 342d983fea894d9f84f50cdc4dc280d6 (table test-table [id=bd4bf1d685254daba2da7aea82263703]) successfully deleted
Sep 24 03:09:07 dist-test-slave-5l98 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219347, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:09:07 dist-test-slave-5l98 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219347, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:07.312125 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:07.296386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38070 (local address 127.0.97.62:37099)
0924 03:09:07.297064 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:07.297068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:07.297089 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:09:07.297144 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:07.297148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:07.297204 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:07.297298 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:07.298735 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.299572 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.300977 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.301390 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.301489 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:07.305778 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:07.305785 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:07.305791 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:07.305840 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:07.309308 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:07.310101 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:07.310105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:07.310107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:07.310253 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:07.310678 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:07.310682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:07.310685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:07.311070 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:07.311276 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:07.311692 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 03:09:07.311871 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":374,"threads_started":1}
I20260924 03:09:07.315865 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:09:07.316461 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:07.323082 2276 tablet_service.cc:1467] Processing CreateTablet for tablet a413d6ab9a4a41788638b26aacd435e0 (DEFAULT_TABLE table=test-table [id=4c4b1502dc8d4eda9bb532431993a452]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:07.323269 2415 tablet_service.cc:1467] Processing CreateTablet for tablet a413d6ab9a4a41788638b26aacd435e0 (DEFAULT_TABLE table=test-table [id=4c4b1502dc8d4eda9bb532431993a452]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:07.323519 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a413d6ab9a4a41788638b26aacd435e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:07.323635 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a413d6ab9a4a41788638b26aacd435e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:07.323660 2554 tablet_service.cc:1467] Processing CreateTablet for tablet a413d6ab9a4a41788638b26aacd435e0 (DEFAULT_TABLE table=test-table [id=4c4b1502dc8d4eda9bb532431993a452]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:07.324115 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a413d6ab9a4a41788638b26aacd435e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:07.330192 2743 tablet_bootstrap.cc:492] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303: Bootstrap starting.
I20260924 03:09:07.330886 2745 tablet_bootstrap.cc:492] T a413d6ab9a4a41788638b26aacd435e0 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Bootstrap starting.
I20260924 03:09:07.331712 2744 tablet_bootstrap.cc:492] T a413d6ab9a4a41788638b26aacd435e0 P b3b67e6f90164055b29ca57d13548479: Bootstrap starting.
I20260924 03:09:07.332700 2743 tablet_bootstrap.cc:654] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:07.332844 2745 tablet_bootstrap.cc:654] T a413d6ab9a4a41788638b26aacd435e0 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:07.333451 2744 tablet_bootstrap.cc:654] T a413d6ab9a4a41788638b26aacd435e0 P b3b67e6f90164055b29ca57d13548479: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:07.335839 2744 tablet_bootstrap.cc:492] T a413d6ab9a4a41788638b26aacd435e0 P b3b67e6f90164055b29ca57d13548479: No bootstrap required, opened a new log
I20260924 03:09:07.336010 2744 ts_tablet_manager.cc:1397] T a413d6ab9a4a41788638b26aacd435e0 P b3b67e6f90164055b29ca57d13548479: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20260924 03:09:07.336843 2744 raft_consensus.cc:348] T a413d6ab9a4a41788638b26aacd435e0 P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:07.337108 2744 raft_consensus.cc:374] T a413d6ab9a4a41788638b26aacd435e0 P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:07.337198 2744 raft_consensus.cc:729] T a413d6ab9a4a41788638b26aacd435e0 P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b67e6f90164055b29ca57d13548479, State: Initialized, Role: FOLLOWER
I20260924 03:09:07.337430 2744 consensus_queue.cc:260] T a413d6ab9a4a41788638b26aacd435e0 P b3b67e6f90164055b29ca57d13548479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:07.338243 2744 ts_tablet_manager.cc:1428] T a413d6ab9a4a41788638b26aacd435e0 P b3b67e6f90164055b29ca57d13548479: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:09:07.341668 2743 tablet_bootstrap.cc:492] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303: No bootstrap required, opened a new log
I20260924 03:09:07.341851 2743 ts_tablet_manager.cc:1397] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260924 03:09:07.342736 2745 tablet_bootstrap.cc:492] T a413d6ab9a4a41788638b26aacd435e0 P 1c77b6c7f2fd44fcb8e81719b50e9b88: No bootstrap required, opened a new log
I20260924 03:09:07.342715 2743 raft_consensus.cc:348] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:07.342913 2745 ts_tablet_manager.cc:1397] T a413d6ab9a4a41788638b26aacd435e0 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260924 03:09:07.342940 2743 raft_consensus.cc:374] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:07.343084 2743 raft_consensus.cc:729] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff49ff7c1f3c46deb3d651443b6f8303, State: Initialized, Role: FOLLOWER
I20260924 03:09:07.343370 2743 consensus_queue.cc:260] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:07.343713 2745 raft_consensus.cc:348] T a413d6ab9a4a41788638b26aacd435e0 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:07.343971 2745 raft_consensus.cc:374] T a413d6ab9a4a41788638b26aacd435e0 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:07.343989 2743 ts_tablet_manager.cc:1428] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 03:09:07.344074 2745 raft_consensus.cc:729] T a413d6ab9a4a41788638b26aacd435e0 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c77b6c7f2fd44fcb8e81719b50e9b88, State: Initialized, Role: FOLLOWER
I20260924 03:09:07.344375 2745 consensus_queue.cc:260] T a413d6ab9a4a41788638b26aacd435e0 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:07.345026 2745 ts_tablet_manager.cc:1428] T a413d6ab9a4a41788638b26aacd435e0 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:09:07.392165 2727 raft_consensus.cc:482] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:07.392376 2727 raft_consensus.cc:504] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:07.393082 2727 leader_election.cc:288] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c77b6c7f2fd44fcb8e81719b50e9b88 (127.0.97.1:42511), b3b67e6f90164055b29ca57d13548479 (127.0.97.2:44791)
I20260924 03:09:07.393687 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a413d6ab9a4a41788638b26aacd435e0" candidate_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" is_pre_election: true
I20260924 03:09:07.393812 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a413d6ab9a4a41788638b26aacd435e0" candidate_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b67e6f90164055b29ca57d13548479" is_pre_election: true
I20260924 03:09:07.393951 2301 raft_consensus.cc:2454] T a413d6ab9a4a41788638b26aacd435e0 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff49ff7c1f3c46deb3d651443b6f8303 in term 0.
I20260924 03:09:07.394098 2438 raft_consensus.cc:2454] T a413d6ab9a4a41788638b26aacd435e0 P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff49ff7c1f3c46deb3d651443b6f8303 in term 0.
I20260924 03:09:07.394454 2506 leader_election.cc:302] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c77b6c7f2fd44fcb8e81719b50e9b88, ff49ff7c1f3c46deb3d651443b6f8303; no voters:
I20260924 03:09:07.394763 2727 raft_consensus.cc:2781] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:07.394897 2727 raft_consensus.cc:482] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:07.395061 2727 raft_consensus.cc:3037] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:07.397665 2727 raft_consensus.cc:504] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:07.398201 2727 leader_election.cc:288] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 [CANDIDATE]: Term 1 election: Requested vote from peers 1c77b6c7f2fd44fcb8e81719b50e9b88 (127.0.97.1:42511), b3b67e6f90164055b29ca57d13548479 (127.0.97.2:44791)
I20260924 03:09:07.398826 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a413d6ab9a4a41788638b26aacd435e0" candidate_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88"
I20260924 03:09:07.398939 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a413d6ab9a4a41788638b26aacd435e0" candidate_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b67e6f90164055b29ca57d13548479"
I20260924 03:09:07.399065 2301 raft_consensus.cc:3037] T a413d6ab9a4a41788638b26aacd435e0 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:07.399153 2438 raft_consensus.cc:3037] T a413d6ab9a4a41788638b26aacd435e0 P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:07.401269 2438 raft_consensus.cc:2454] T a413d6ab9a4a41788638b26aacd435e0 P b3b67e6f90164055b29ca57d13548479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff49ff7c1f3c46deb3d651443b6f8303 in term 1.
I20260924 03:09:07.401302 2301 raft_consensus.cc:2454] T a413d6ab9a4a41788638b26aacd435e0 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff49ff7c1f3c46deb3d651443b6f8303 in term 1.
I20260924 03:09:07.401700 2506 leader_election.cc:302] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3b67e6f90164055b29ca57d13548479, ff49ff7c1f3c46deb3d651443b6f8303; no voters:
I20260924 03:09:07.402006 2727 raft_consensus.cc:2781] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:07.402192 2727 raft_consensus.cc:686] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 [term 1 LEADER]: Becoming Leader. State: Replica: ff49ff7c1f3c46deb3d651443b6f8303, State: Running, Role: LEADER
I20260924 03:09:07.402498 2727 consensus_queue.cc:237] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:07.405489 2139 catalog_manager.cc:5495] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff49ff7c1f3c46deb3d651443b6f8303 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } health_report { overall_health: HEALTHY } } }
I20260924 03:09:07.444840 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:07.439311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49426 (local address 127.0.97.3:40409)
0924 03:09:07.439965 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:07.439970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:07.439988 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:09:07.440041 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:07.440044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:07.440114 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:07.440231 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:07.440712 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.441598 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.442979 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.443434 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.443591 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:07.444016 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:07.444119 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:07.444226 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:07.444394 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 03:09:07.444473 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":328,"threads_started":1}
I20260924 03:09:07.459319 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:07.454272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33613 (local address 127.0.97.3:40409)
0924 03:09:07.454463 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:07.454472 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:07.454606 (+ 134us) server_negotiation.cc:407] Connection header received
0924 03:09:07.454814 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:07.454820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:07.454894 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:07.454999 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:07.455464 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.456224 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.457680 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.458085 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.458230 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:07.458615 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:07.458705 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:07.458804 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:07.458911 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 03:09:07.458982 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":58}
I20260924 03:09:07.459385 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:07.453907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40409 (local address 127.0.97.3:33613)
0924 03:09:07.454500 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:07.454518 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:07.454689 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:07.455009 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:07.455013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:07.455024 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:09:07.455267 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:07.455275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.456374 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.456381 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:07.457537 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:07.457548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.458264 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.458270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:07.458427 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:07.458460 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:09:07.458802 (+ 342us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:09:07.458813 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 03:09:07.458895 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 03:09:07.459020 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":303,"threads_started":1}
I20260924 03:09:07.467876 2301 raft_consensus.cc:1261] T a413d6ab9a4a41788638b26aacd435e0 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 1 FOLLOWER]: Refusing update from remote peer ff49ff7c1f3c46deb3d651443b6f8303: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:07.467994 2438 raft_consensus.cc:1261] T a413d6ab9a4a41788638b26aacd435e0 P b3b67e6f90164055b29ca57d13548479 [term 1 FOLLOWER]: Refusing update from remote peer ff49ff7c1f3c46deb3d651443b6f8303: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:07.468528 2727 consensus_queue.cc:1035] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:07.468900 2712 consensus_queue.cc:1035] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:07.519807 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:07.520068 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:07.523217 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9: Sending DeleteTablet for 3 replicas of tablet a413d6ab9a4a41788638b26aacd435e0
I20260924 03:09:07.524008 2276 tablet_service.cc:1514] Processing DeleteTablet for tablet a413d6ab9a4a41788638b26aacd435e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35090
I20260924 03:09:07.524291 2732 tablet_replica.cc:330] T a413d6ab9a4a41788638b26aacd435e0 P 1c77b6c7f2fd44fcb8e81719b50e9b88: stopping tablet replica
I20260924 03:09:07.524247 2415 tablet_service.cc:1514] Processing DeleteTablet for tablet a413d6ab9a4a41788638b26aacd435e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55266
I20260924 03:09:07.524534 2732 raft_consensus.cc:2229] T a413d6ab9a4a41788638b26aacd435e0 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:07.524739 2732 raft_consensus.cc:2258] T a413d6ab9a4a41788638b26aacd435e0 P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:07.525027 2734 tablet_replica.cc:330] T a413d6ab9a4a41788638b26aacd435e0 P b3b67e6f90164055b29ca57d13548479: stopping tablet replica
I20260924 03:09:07.525234 2734 raft_consensus.cc:2229] T a413d6ab9a4a41788638b26aacd435e0 P b3b67e6f90164055b29ca57d13548479 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:07.525410 2554 tablet_service.cc:1514] Processing DeleteTablet for tablet a413d6ab9a4a41788638b26aacd435e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59700
I20260924 03:09:07.525460 2734 raft_consensus.cc:2258] T a413d6ab9a4a41788638b26aacd435e0 P b3b67e6f90164055b29ca57d13548479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:07.525754 2733 tablet_replica.cc:330] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303: stopping tablet replica
I20260924 03:09:07.525959 2733 raft_consensus.cc:2229] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:09:07.526011 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 03:09:07.526492 2733 raft_consensus.cc:2258] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 03:09:07.526702 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:07.527515 2732 ts_tablet_manager.cc:1905] T a413d6ab9a4a41788638b26aacd435e0 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:07.527750 2733 ts_tablet_manager.cc:1905] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:07.534418 2276 tablet_service.cc:1467] Processing CreateTablet for tablet f3a3b3fb5f264d5cad341462dfe0e38c (DEFAULT_TABLE table=test-table [id=aee2fba569624daf847d112702e4cc86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:07.534540 2415 tablet_service.cc:1467] Processing CreateTablet for tablet f3a3b3fb5f264d5cad341462dfe0e38c (DEFAULT_TABLE table=test-table [id=aee2fba569624daf847d112702e4cc86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:07.534912 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3a3b3fb5f264d5cad341462dfe0e38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:07.534992 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3a3b3fb5f264d5cad341462dfe0e38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:07.536726 2554 tablet_service.cc:1467] Processing CreateTablet for tablet f3a3b3fb5f264d5cad341462dfe0e38c (DEFAULT_TABLE table=test-table [id=aee2fba569624daf847d112702e4cc86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:07.537142 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3a3b3fb5f264d5cad341462dfe0e38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:07.541234 2744 tablet_bootstrap.cc:492] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479: Bootstrap starting.
I20260924 03:09:07.542164 2745 tablet_bootstrap.cc:492] T f3a3b3fb5f264d5cad341462dfe0e38c P 1c77b6c7f2fd44fcb8e81719b50e9b88: Bootstrap starting.
I20260924 03:09:07.542842 2744 tablet_bootstrap.cc:654] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:07.541236 2734 ts_tablet_manager.cc:1905] T a413d6ab9a4a41788638b26aacd435e0 P b3b67e6f90164055b29ca57d13548479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:07.544138 2745 tablet_bootstrap.cc:654] T f3a3b3fb5f264d5cad341462dfe0e38c P 1c77b6c7f2fd44fcb8e81719b50e9b88: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:07.544921 2733 ts_tablet_manager.cc:1918] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:09:07.547122 2733 log.cc:1196] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/a413d6ab9a4a41788638b26aacd435e0
I20260924 03:09:07.548475 2732 ts_tablet_manager.cc:1918] T a413d6ab9a4a41788638b26aacd435e0 P 1c77b6c7f2fd44fcb8e81719b50e9b88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:09:07.548676 2732 log.cc:1196] T a413d6ab9a4a41788638b26aacd435e0 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/a413d6ab9a4a41788638b26aacd435e0
I20260924 03:09:07.549134 2732 ts_tablet_manager.cc:1939] T a413d6ab9a4a41788638b26aacd435e0 P 1c77b6c7f2fd44fcb8e81719b50e9b88: Deleting consensus metadata
I20260924 03:09:07.549808 2733 ts_tablet_manager.cc:1939] T a413d6ab9a4a41788638b26aacd435e0 P ff49ff7c1f3c46deb3d651443b6f8303: Deleting consensus metadata
I20260924 03:09:07.550061 2734 ts_tablet_manager.cc:1918] T a413d6ab9a4a41788638b26aacd435e0 P b3b67e6f90164055b29ca57d13548479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:09:07.550204 2734 log.cc:1196] T a413d6ab9a4a41788638b26aacd435e0 P b3b67e6f90164055b29ca57d13548479: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/a413d6ab9a4a41788638b26aacd435e0
I20260924 03:09:07.550148 2124 catalog_manager.cc:4841] TS 1c77b6c7f2fd44fcb8e81719b50e9b88 (127.0.97.1:42511): tablet a413d6ab9a4a41788638b26aacd435e0 (table test-table [id=4c4b1502dc8d4eda9bb532431993a452]) successfully deleted
I20260924 03:09:07.551065 2743 tablet_bootstrap.cc:492] T f3a3b3fb5f264d5cad341462dfe0e38c P ff49ff7c1f3c46deb3d651443b6f8303: Bootstrap starting.
I20260924 03:09:07.553593 2124 catalog_manager.cc:4841] TS ff49ff7c1f3c46deb3d651443b6f8303 (127.0.97.3:40409): tablet a413d6ab9a4a41788638b26aacd435e0 (table test-table [id=4c4b1502dc8d4eda9bb532431993a452]) successfully deleted
I20260924 03:09:07.553840 2743 tablet_bootstrap.cc:654] T f3a3b3fb5f264d5cad341462dfe0e38c P ff49ff7c1f3c46deb3d651443b6f8303: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:07.553999 2734 ts_tablet_manager.cc:1939] T a413d6ab9a4a41788638b26aacd435e0 P b3b67e6f90164055b29ca57d13548479: Deleting consensus metadata
I20260924 03:09:07.555321 2124 catalog_manager.cc:4841] TS b3b67e6f90164055b29ca57d13548479 (127.0.97.2:44791): tablet a413d6ab9a4a41788638b26aacd435e0 (table test-table [id=4c4b1502dc8d4eda9bb532431993a452]) successfully deleted
I20260924 03:09:07.556847 2743 tablet_bootstrap.cc:492] T f3a3b3fb5f264d5cad341462dfe0e38c P ff49ff7c1f3c46deb3d651443b6f8303: No bootstrap required, opened a new log
I20260924 03:09:07.557011 2743 ts_tablet_manager.cc:1397] T f3a3b3fb5f264d5cad341462dfe0e38c P ff49ff7c1f3c46deb3d651443b6f8303: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260924 03:09:07.557207 2744 tablet_bootstrap.cc:492] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479: No bootstrap required, opened a new log
I20260924 03:09:07.557386 2744 ts_tablet_manager.cc:1397] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.003s
I20260924 03:09:07.557483 2745 tablet_bootstrap.cc:492] T f3a3b3fb5f264d5cad341462dfe0e38c P 1c77b6c7f2fd44fcb8e81719b50e9b88: No bootstrap required, opened a new log
I20260924 03:09:07.557647 2745 ts_tablet_manager.cc:1397] T f3a3b3fb5f264d5cad341462dfe0e38c P 1c77b6c7f2fd44fcb8e81719b50e9b88: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.008s
I20260924 03:09:07.557773 2743 raft_consensus.cc:348] T f3a3b3fb5f264d5cad341462dfe0e38c P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:07.558018 2743 raft_consensus.cc:374] T f3a3b3fb5f264d5cad341462dfe0e38c P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:07.558101 2743 raft_consensus.cc:729] T f3a3b3fb5f264d5cad341462dfe0e38c P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff49ff7c1f3c46deb3d651443b6f8303, State: Initialized, Role: FOLLOWER
I20260924 03:09:07.558089 2744 raft_consensus.cc:348] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:07.558372 2744 raft_consensus.cc:374] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:07.558476 2744 raft_consensus.cc:729] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b67e6f90164055b29ca57d13548479, State: Initialized, Role: FOLLOWER
I20260924 03:09:07.558378 2743 consensus_queue.cc:260] T f3a3b3fb5f264d5cad341462dfe0e38c P ff49ff7c1f3c46deb3d651443b6f8303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:07.558414 2745 raft_consensus.cc:348] T f3a3b3fb5f264d5cad341462dfe0e38c P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:07.558631 2745 raft_consensus.cc:374] T f3a3b3fb5f264d5cad341462dfe0e38c P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:07.558710 2745 raft_consensus.cc:729] T f3a3b3fb5f264d5cad341462dfe0e38c P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c77b6c7f2fd44fcb8e81719b50e9b88, State: Initialized, Role: FOLLOWER
I20260924 03:09:07.558703 2744 consensus_queue.cc:260] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:07.558933 2745 consensus_queue.cc:260] T f3a3b3fb5f264d5cad341462dfe0e38c P 1c77b6c7f2fd44fcb8e81719b50e9b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:07.559278 2744 ts_tablet_manager.cc:1428] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:09:07.559543 2745 ts_tablet_manager.cc:1428] T f3a3b3fb5f264d5cad341462dfe0e38c P 1c77b6c7f2fd44fcb8e81719b50e9b88: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 03:09:07.561481 2743 ts_tablet_manager.cc:1428] T f3a3b3fb5f264d5cad341462dfe0e38c P ff49ff7c1f3c46deb3d651443b6f8303: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 03:09:07.613049 2749 raft_consensus.cc:482] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:07.613260 2749 raft_consensus.cc:504] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:07.614104 2749 leader_election.cc:288] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c77b6c7f2fd44fcb8e81719b50e9b88 (127.0.97.1:42511), ff49ff7c1f3c46deb3d651443b6f8303 (127.0.97.3:40409)
I20260924 03:09:07.614653 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a3b3fb5f264d5cad341462dfe0e38c" candidate_uuid: "b3b67e6f90164055b29ca57d13548479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" is_pre_election: true
I20260924 03:09:07.614769 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a3b3fb5f264d5cad341462dfe0e38c" candidate_uuid: "b3b67e6f90164055b29ca57d13548479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" is_pre_election: true
I20260924 03:09:07.614905 2301 raft_consensus.cc:2454] T f3a3b3fb5f264d5cad341462dfe0e38c P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b67e6f90164055b29ca57d13548479 in term 0.
I20260924 03:09:07.615061 2579 raft_consensus.cc:2454] T f3a3b3fb5f264d5cad341462dfe0e38c P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b67e6f90164055b29ca57d13548479 in term 0.
I20260924 03:09:07.615343 2367 leader_election.cc:302] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c77b6c7f2fd44fcb8e81719b50e9b88, b3b67e6f90164055b29ca57d13548479; no voters:
I20260924 03:09:07.615623 2749 raft_consensus.cc:2781] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:07.615746 2749 raft_consensus.cc:482] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:07.615828 2749 raft_consensus.cc:3037] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:07.618032 2749 raft_consensus.cc:504] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:07.618598 2749 leader_election.cc:288] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 [CANDIDATE]: Term 1 election: Requested vote from peers 1c77b6c7f2fd44fcb8e81719b50e9b88 (127.0.97.1:42511), ff49ff7c1f3c46deb3d651443b6f8303 (127.0.97.3:40409)
I20260924 03:09:07.619077 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a3b3fb5f264d5cad341462dfe0e38c" candidate_uuid: "b3b67e6f90164055b29ca57d13548479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff49ff7c1f3c46deb3d651443b6f8303"
I20260924 03:09:07.619113 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a3b3fb5f264d5cad341462dfe0e38c" candidate_uuid: "b3b67e6f90164055b29ca57d13548479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88"
I20260924 03:09:07.619294 2579 raft_consensus.cc:3037] T f3a3b3fb5f264d5cad341462dfe0e38c P ff49ff7c1f3c46deb3d651443b6f8303 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:07.619434 2301 raft_consensus.cc:3037] T f3a3b3fb5f264d5cad341462dfe0e38c P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:07.622876 2579 raft_consensus.cc:2454] T f3a3b3fb5f264d5cad341462dfe0e38c P ff49ff7c1f3c46deb3d651443b6f8303 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b67e6f90164055b29ca57d13548479 in term 1.
I20260924 03:09:07.622865 2301 raft_consensus.cc:2454] T f3a3b3fb5f264d5cad341462dfe0e38c P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b67e6f90164055b29ca57d13548479 in term 1.
I20260924 03:09:07.623327 2367 leader_election.cc:302] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3b67e6f90164055b29ca57d13548479, ff49ff7c1f3c46deb3d651443b6f8303; no voters:
I20260924 03:09:07.623835 2749 raft_consensus.cc:2781] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:07.624298 2749 raft_consensus.cc:686] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 [term 1 LEADER]: Becoming Leader. State: Replica: b3b67e6f90164055b29ca57d13548479, State: Running, Role: LEADER
I20260924 03:09:07.624629 2749 consensus_queue.cc:237] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } }
I20260924 03:09:07.627523 2137 catalog_manager.cc:5495] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b67e6f90164055b29ca57d13548479 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b3b67e6f90164055b29ca57d13548479" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b67e6f90164055b29ca57d13548479" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 } health_report { overall_health: UNKNOWN } } }
I20260924 03:09:07.670678 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:07.660882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34688 (local address 127.0.97.2:44791)
0924 03:09:07.665676 (+ 4794us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:07.665680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:07.665700 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:09:07.665755 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:07.665759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:07.665866 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:07.666004 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:07.666440 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.667366 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.668828 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.669262 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.669397 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:07.669849 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:07.669962 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:07.670066 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:07.670214 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 03:09:07.670288 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4668,"thread_start_us":4551,"threads_started":1}
I20260924 03:09:07.680454 2301 raft_consensus.cc:1261] T f3a3b3fb5f264d5cad341462dfe0e38c P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 1 FOLLOWER]: Refusing update from remote peer b3b67e6f90164055b29ca57d13548479: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:07.680622 2579 raft_consensus.cc:1261] T f3a3b3fb5f264d5cad341462dfe0e38c P ff49ff7c1f3c46deb3d651443b6f8303 [term 1 FOLLOWER]: Refusing update from remote peer b3b67e6f90164055b29ca57d13548479: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:07.681130 2762 consensus_queue.cc:1035] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff49ff7c1f3c46deb3d651443b6f8303" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:07.681491 2749 consensus_queue.cc:1035] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c77b6c7f2fd44fcb8e81719b50e9b88" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 03:09:07.707713 2713 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 03:09:09.300613 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:09.300822 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:09.305307 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffa7ec1169d7481cb599370d0f9f62f9: Sending DeleteTablet for 3 replicas of tablet f3a3b3fb5f264d5cad341462dfe0e38c
I20260924 03:09:09.306195 2276 tablet_service.cc:1514] Processing DeleteTablet for tablet f3a3b3fb5f264d5cad341462dfe0e38c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:09 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35090
I20260924 03:09:09.306339 2415 tablet_service.cc:1514] Processing DeleteTablet for tablet f3a3b3fb5f264d5cad341462dfe0e38c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:09 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55266
I20260924 03:09:09.306337 2554 tablet_service.cc:1514] Processing DeleteTablet for tablet f3a3b3fb5f264d5cad341462dfe0e38c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:09 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59700
I20260924 03:09:09.307268 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 2210
I20260924 03:09:09.315872 2792 tablet_replica.cc:330] T f3a3b3fb5f264d5cad341462dfe0e38c P 1c77b6c7f2fd44fcb8e81719b50e9b88: stopping tablet replica
I20260924 03:09:09.316179 2792 raft_consensus.cc:2229] T f3a3b3fb5f264d5cad341462dfe0e38c P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:09.316562 2792 raft_consensus.cc:2258] T f3a3b3fb5f264d5cad341462dfe0e38c P 1c77b6c7f2fd44fcb8e81719b50e9b88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:09.317787 2792 ts_tablet_manager.cc:1905] T f3a3b3fb5f264d5cad341462dfe0e38c P 1c77b6c7f2fd44fcb8e81719b50e9b88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:09.319069 2794 tablet_replica.cc:330] T f3a3b3fb5f264d5cad341462dfe0e38c P ff49ff7c1f3c46deb3d651443b6f8303: stopping tablet replica
I20260924 03:09:09.319334 2794 raft_consensus.cc:2229] T f3a3b3fb5f264d5cad341462dfe0e38c P ff49ff7c1f3c46deb3d651443b6f8303 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:09.319643 2794 raft_consensus.cc:2258] T f3a3b3fb5f264d5cad341462dfe0e38c P ff49ff7c1f3c46deb3d651443b6f8303 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:09.321063 2794 ts_tablet_manager.cc:1905] T f3a3b3fb5f264d5cad341462dfe0e38c P ff49ff7c1f3c46deb3d651443b6f8303: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:09.323202 2793 tablet_replica.cc:330] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479: stopping tablet replica
I20260924 03:09:09.323419 2793 raft_consensus.cc:2229] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:09:09.323812 2793 raft_consensus.cc:2258] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:09.324766 2793 ts_tablet_manager.cc:1905] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:09.325714 2792 ts_tablet_manager.cc:1918] T f3a3b3fb5f264d5cad341462dfe0e38c P 1c77b6c7f2fd44fcb8e81719b50e9b88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:09:09.325851 2792 log.cc:1196] T f3a3b3fb5f264d5cad341462dfe0e38c P 1c77b6c7f2fd44fcb8e81719b50e9b88: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/f3a3b3fb5f264d5cad341462dfe0e38c
I20260924 03:09:09.326304 2792 ts_tablet_manager.cc:1939] T f3a3b3fb5f264d5cad341462dfe0e38c P 1c77b6c7f2fd44fcb8e81719b50e9b88: Deleting consensus metadata
I20260924 03:09:09.326932 2794 ts_tablet_manager.cc:1918] T f3a3b3fb5f264d5cad341462dfe0e38c P ff49ff7c1f3c46deb3d651443b6f8303: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:09:09.327069 2794 log.cc:1196] T f3a3b3fb5f264d5cad341462dfe0e38c P ff49ff7c1f3c46deb3d651443b6f8303: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/f3a3b3fb5f264d5cad341462dfe0e38c
I20260924 03:09:09.327447 2794 ts_tablet_manager.cc:1939] T f3a3b3fb5f264d5cad341462dfe0e38c P ff49ff7c1f3c46deb3d651443b6f8303: Deleting consensus metadata
I20260924 03:09:09.327390 2124 catalog_manager.cc:4841] TS 1c77b6c7f2fd44fcb8e81719b50e9b88 (127.0.97.1:42511): tablet f3a3b3fb5f264d5cad341462dfe0e38c (table test-table [id=aee2fba569624daf847d112702e4cc86]) successfully deleted
I20260924 03:09:09.328541 2124 catalog_manager.cc:4841] TS ff49ff7c1f3c46deb3d651443b6f8303 (127.0.97.3:40409): tablet f3a3b3fb5f264d5cad341462dfe0e38c (table test-table [id=aee2fba569624daf847d112702e4cc86]) successfully deleted
I20260924 03:09:09.330734 2793 ts_tablet_manager.cc:1918] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:09:09.330933 2793 log.cc:1196] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/f3a3b3fb5f264d5cad341462dfe0e38c
I20260924 03:09:09.331354 2793 ts_tablet_manager.cc:1939] T f3a3b3fb5f264d5cad341462dfe0e38c P b3b67e6f90164055b29ca57d13548479: Deleting consensus metadata
Sep 24 03:09:09 dist-test-slave-5l98 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219347, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:09:09.332333 2124 catalog_manager.cc:4841] TS b3b67e6f90164055b29ca57d13548479 (127.0.97.2:44791): tablet f3a3b3fb5f264d5cad341462dfe0e38c (table test-table [id=aee2fba569624daf847d112702e4cc86]) successfully deleted
I20260924 03:09:09.339918 2796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:09.308962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48358 (local address 127.0.97.1:42511)
0924 03:09:09.315706 (+ 6744us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:09.315717 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:09.315742 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:09:09.315808 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:09.315812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:09.315894 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:09.316006 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:09.317895 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:09.319014 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:09.321337 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:09.321921 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:09.322047 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:09.333227 (+ 11180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:09.333243 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:09.333324 (+ 81us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:09.333371 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:09.336996 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:09.337738 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:09.337745 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:09.337748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:09.337894 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:09.338207 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:09.338210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:09.338212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:09.338578 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:09.338821 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:09.339282 (+ 461us) server_negotiation.cc:299] Negotiation successful
0924 03:09:09.339557 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6539,"thread_start_us":6407,"threads_started":1}
W20260924 03:09:09.340842 2341 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48358
W20260924 03:09:09.341619 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 2210: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:09:09.341881 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 2210
I20260924 03:09:09.367136 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 2350
Sep 24 03:09:09 dist-test-slave-5l98 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219347, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 03:09:09.383704 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:09.367898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34698 (local address 127.0.97.2:44791)
0924 03:09:09.368554 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:09.368564 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:09.368582 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:09:09.368638 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:09.368642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:09.368724 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:09.368835 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:09.370241 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:09.371138 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:09.372422 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:09.373047 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:09.373151 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:09.377452 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:09.377473 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:09.377494 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:09.377538 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:09.380836 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:09.381563 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:09.381572 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:09.381574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:09.381704 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:09.382121 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:09.382125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:09.382127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:09.382542 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:09.382765 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:09.383074 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 03:09:09.383285 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":355,"threads_started":1}
W20260924 03:09:09.384452 2480 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34698
W20260924 03:09:09.385346 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 2350: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:09:09.385555 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 2350
I20260924 03:09:09.409734 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 2489
Sep 24 03:09:09 dist-test-slave-5l98 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219347, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 03:09:09.425985 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:09.410534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49430 (local address 127.0.97.3:40409)
0924 03:09:09.411091 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:09.411096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:09.411110 (+ 14us) server_negotiation.cc:407] Connection header received
0924 03:09:09.411162 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:09.411165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:09.411219 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:09.411343 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:09.412636 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:09.413338 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:09.414666 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:09.415105 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:09.415242 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:09.419709 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:09.419723 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:09.419740 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:09.419787 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:09.422985 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:09.423805 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:09.423813 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:09.423818 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:09.423945 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:09.424356 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:09.424359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:09.424361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:09.424816 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:09.425048 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:09.425490 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 03:09:09.425667 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":278,"threads_started":1}
W20260924 03:09:09.426882 2619 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49430
W20260924 03:09:09.427655 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 2489: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:09:09.427863 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 2489
I20260924 03:09:09.453450 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 2107
I20260924 03:09:09.454704 2179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:09.571189 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 2107
2026-09-24T03:09:09Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8449 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:09:09 dist-test-slave-5l98 krb5kdc[2802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:09:09 dist-test-slave-5l98 krb5kdc[2802](info): set up 2 sockets
Sep 24 03:09:09 dist-test-slave-5l98 krb5kdc[2802](info): commencing operation
krb5kdc: starting...
W20260924 03:09:10.709453 388 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:09:10 dist-test-slave-5l98 krb5kdc[2802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:09:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:09:10.935845 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34947
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:11.289281 2815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:11.289868 2815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:11.290051 2815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:11.297847 2815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:11.297966 2815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:11.298051 2815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:11.298161 2815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:11.298226 2815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:09:11.308302 2815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:11.309939 2815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:11.311306 2815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:11.325500 2820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:11.325500 2823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:11.325577 2821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:11.325841 2815 server_base.cc:1034] running on GCE node
I20260924 03:09:11.326948 2815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:11.328435 2815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:11.329746 2815 hybrid_clock.cc:648] HybridClock initialized: now 1790219351329671 us; error 56 us; skew 500 ppm
I20260924 03:09:11.330170 2815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:11 dist-test-slave-5l98 krb5kdc[2802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:11.334059 2815 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:09:11.336185 2815 webserver.cc:460] Webserver started at http://127.0.97.62:44143/ using document root <none> and password file <none>
I20260924 03:09:11.337028 2815 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:11.337195 2815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:11.337523 2815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:11.340699 2815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "3585256f7c614b698affc298985e9da7"
format_stamp: "Formatted at 2026-09-24 03:09:11 on dist-test-slave-5l98"
server_key: "3465a4a363afe99add7149fbce39f00d"
server_key_iv: "f298e4ec3bc8dc948becae81f145b9ac"
server_key_version: "encryptionkey@0"
I20260924 03:09:11.341611 2815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "3585256f7c614b698affc298985e9da7"
format_stamp: "Formatted at 2026-09-24 03:09:11 on dist-test-slave-5l98"
server_key: "3465a4a363afe99add7149fbce39f00d"
server_key_iv: "f298e4ec3bc8dc948becae81f145b9ac"
server_key_version: "encryptionkey@0"
I20260924 03:09:11.347875 2815 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260924 03:09:11.353130 2830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:11.355201 2815 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260924 03:09:11.355448 2815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "3585256f7c614b698affc298985e9da7"
format_stamp: "Formatted at 2026-09-24 03:09:11 on dist-test-slave-5l98"
server_key: "3465a4a363afe99add7149fbce39f00d"
server_key_iv: "f298e4ec3bc8dc948becae81f145b9ac"
server_key_version: "encryptionkey@0"
I20260924 03:09:11.355674 2815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:11.370406 2815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:11.375126 2815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:11.375443 2815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:11.395399 2815 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34947
I20260924 03:09:11.395453 2881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34947 every 8 connection(s)
I20260924 03:09:11.396837 2815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:09:11.397445 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 2815
I20260924 03:09:11.397703 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:09:11.398458 388 external_mini_cluster.cc:1428] Setting key 1e4f8e894985c3b0f75b63d1e413da27
I20260924 03:09:11.401568 2882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 03:09:11 dist-test-slave-5l98 krb5kdc[2802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:11.415882 2882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7: Bootstrap starting.
I20260924 03:09:11.419257 2882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:11.420320 2882 log.cc:824] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:11.423369 2882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7: No bootstrap required, opened a new log
I20260924 03:09:11.423868 2885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:11.400668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44000 (local address 127.0.97.62:34947)
0924 03:09:11.401876 (+ 1208us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:11.401889 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:11.401935 (+ 46us) server_negotiation.cc:407] Connection header received
0924 03:09:11.403544 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:11.403566 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:11.403987 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:11.404352 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:11.406807 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:11.408276 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:11.410107 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:11.410835 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:11.411078 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:11.416199 (+ 5121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:11.416208 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:11.416224 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:11.416267 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:11.419617 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:11.420349 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:11.420357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:11.420363 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:11.420619 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:11.421046 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:11.421051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:11.421058 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:11.421585 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:11.421898 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:11.422307 (+ 409us) server_negotiation.cc:299] Negotiation successful
0924 03:09:11.422755 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":389,"threads_started":1}
I20260924 03:09:11.431810 2882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3585256f7c614b698affc298985e9da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34947 } }
I20260924 03:09:11.432222 2882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:11.432325 2882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3585256f7c614b698affc298985e9da7, State: Initialized, Role: FOLLOWER
I20260924 03:09:11.433132 2882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3585256f7c614b698affc298985e9da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34947 } }
I20260924 03:09:11.433374 2882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:09:11.433522 2882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:09:11.433709 2882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:11.435889 2882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3585256f7c614b698affc298985e9da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34947 } }
I20260924 03:09:11.436420 2882 leader_election.cc:302] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3585256f7c614b698affc298985e9da7; no voters:
I20260924 03:09:11.437162 2882 leader_election.cc:288] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:09:11.437326 2887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:11.438810 2887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7 [term 1 LEADER]: Becoming Leader. State: Replica: 3585256f7c614b698affc298985e9da7, State: Running, Role: LEADER
I20260924 03:09:11.439348 2887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3585256f7c614b698affc298985e9da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34947 } }
I20260924 03:09:11.439656 2882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:09:11.442660 2889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3585256f7c614b698affc298985e9da7. Latest consensus state: current_term: 1 leader_uuid: "3585256f7c614b698affc298985e9da7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3585256f7c614b698affc298985e9da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34947 } } }
I20260924 03:09:11.442740 2888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3585256f7c614b698affc298985e9da7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3585256f7c614b698affc298985e9da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34947 } } }
I20260924 03:09:11.443106 2889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:11.443172 2888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:11.444085 2895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:09:11.450965 2895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:09:11.461678 2895 catalog_manager.cc:1347] Generated new cluster ID: 3a8e50caa1c34110b57025aa332b127d
I20260924 03:09:11.462010 2895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:09:11.479954 2895 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:09:11.481755 2895 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:09:11.495555 2895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3585256f7c614b698affc298985e9da7: Generated new TSK 0
I20260924 03:09:11.496542 2895 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 03:09:11.553157 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34947
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:11.921146 2908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:11.921429 2908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:11.921522 2908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:11.929032 2908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:11.929149 2908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:11.929262 2908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:09:11.939314 2908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:11.940815 2908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:11.942272 2908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:11.956470 2913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:11.956522 2914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:11.956655 2916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:11.957858 2908 server_base.cc:1034] running on GCE node
I20260924 03:09:11.958588 2908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:11.959722 2908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:11.961090 2908 hybrid_clock.cc:648] HybridClock initialized: now 1790219351961023 us; error 64 us; skew 500 ppm
I20260924 03:09:11.961455 2908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:11 dist-test-slave-5l98 krb5kdc[2802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:11.965083 2908 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:09:11.966653 2908 webserver.cc:460] Webserver started at http://127.0.97.1:41707/ using document root <none> and password file <none>
I20260924 03:09:11.967396 2908 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:11.967501 2908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:11.967830 2908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:11.970881 2908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "e27e2e145bae466c8e9e0630068f1f03"
format_stamp: "Formatted at 2026-09-24 03:09:11 on dist-test-slave-5l98"
server_key: "63bdf2eb9b58a7325b5e561491edb28a"
server_key_iv: "64daf44f375bbffcf820607ac93711a0"
server_key_version: "encryptionkey@0"
I20260924 03:09:11.971700 2908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e27e2e145bae466c8e9e0630068f1f03"
format_stamp: "Formatted at 2026-09-24 03:09:11 on dist-test-slave-5l98"
server_key: "63bdf2eb9b58a7325b5e561491edb28a"
server_key_iv: "64daf44f375bbffcf820607ac93711a0"
server_key_version: "encryptionkey@0"
I20260924 03:09:11.977172 2908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 03:09:11.980926 2923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:11.982306 2908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:09:11.982508 2908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "e27e2e145bae466c8e9e0630068f1f03"
format_stamp: "Formatted at 2026-09-24 03:09:11 on dist-test-slave-5l98"
server_key: "63bdf2eb9b58a7325b5e561491edb28a"
server_key_iv: "64daf44f375bbffcf820607ac93711a0"
server_key_version: "encryptionkey@0"
I20260924 03:09:11.982687 2908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:12.012647 2908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:12.017794 2908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:12.018112 2908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:12.019114 2908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:12.020633 2908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:12.020706 2908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:12.020823 2908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:12.020910 2908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:12.057777 2908 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42003
I20260924 03:09:12.057827 3035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42003 every 8 connection(s)
I20260924 03:09:12.059099 2908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:09:12.067086 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 2908
I20260924 03:09:12.067325 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:09:12.067961 388 external_mini_cluster.cc:1428] Setting key 4997d8c1b1728d1871747c3ebbc798a0
Sep 24 03:09:12 dist-test-slave-5l98 krb5kdc[2802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:12.081862 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:12.062364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43553 (local address 127.0.97.62:34947)
0924 03:09:12.062918 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:12.062923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:12.064070 (+ 1147us) server_negotiation.cc:407] Connection header received
0924 03:09:12.066099 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:12.066104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:12.066199 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:12.066324 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:12.067999 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:12.068953 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:12.070780 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:12.071198 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:12.071340 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:12.075350 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:12.075356 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:12.075359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:12.075391 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:12.078172 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:12.079181 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:12.079185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:12.079187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:12.079304 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:12.079841 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:12.079845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:12.079847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:12.080271 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:12.080394 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:12.081396 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0924 03:09:12.081577 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":273,"threads_started":1}
I20260924 03:09:12.083499 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:12.063122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34947 (local address 127.0.97.1:43553)
0924 03:09:12.063876 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:12.063921 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:12.065797 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:12.066577 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:12.066586 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:12.067126 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:12.067815 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:12.067833 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:12.069150 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:12.069154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:12.070611 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:12.070628 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:12.071446 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:12.071455 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:12.071712 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:12.072604 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:12.072628 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:12.075079 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:12.078332 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:12.078345 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:12.078367 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:12.079027 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:12.079433 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:12.079438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:12.079444 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:12.079702 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:12.080415 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:12.080426 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:12.080927 (+ 501us) client_negotiation.cc:769] Sending connection context
0924 03:09:12.081282 (+ 355us) client_negotiation.cc:240] Negotiation successful
0924 03:09:12.081894 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":333,"threads_started":1}
I20260924 03:09:12.085390 3036 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34947
I20260924 03:09:12.085752 3036 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:12.086733 3036 heartbeater.cc:511] Master 127.0.97.62:34947 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 03:09:12.089432 2847 ts_manager.cc:194] Registered new tserver with Master: e27e2e145bae466c8e9e0630068f1f03 (127.0.97.1:42003)
I20260924 03:09:12.091094 2847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 03:09:12.118271 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34947
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:12.457223 3042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:12.457512 3042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:12.457665 3042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:12.465571 3042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:12.465745 3042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:12.465943 3042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:09:12.476228 3042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:12.477659 3042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:12.478979 3042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:12.493142 3047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:12.493430 3048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:12.493630 3050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:12.494741 3042 server_base.cc:1034] running on GCE node
I20260924 03:09:12.495522 3042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:12.496662 3042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:12.497882 3042 hybrid_clock.cc:648] HybridClock initialized: now 1790219352497833 us; error 44 us; skew 500 ppm
I20260924 03:09:12.498322 3042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:12 dist-test-slave-5l98 krb5kdc[2802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:12.501722 3042 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:09:12.503283 3042 webserver.cc:460] Webserver started at http://127.0.97.2:42001/ using document root <none> and password file <none>
I20260924 03:09:12.503969 3042 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:12.504073 3042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:12.504324 3042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:12.507443 3042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "0d9642458c724bbda81925f999e163ca"
format_stamp: "Formatted at 2026-09-24 03:09:12 on dist-test-slave-5l98"
server_key: "571c4ae5a4dfba8a6cde34d347173fc4"
server_key_iv: "a0a750bbd5282bed77bb724b56d36dcd"
server_key_version: "encryptionkey@0"
I20260924 03:09:12.508199 3042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0d9642458c724bbda81925f999e163ca"
format_stamp: "Formatted at 2026-09-24 03:09:12 on dist-test-slave-5l98"
server_key: "571c4ae5a4dfba8a6cde34d347173fc4"
server_key_iv: "a0a750bbd5282bed77bb724b56d36dcd"
server_key_version: "encryptionkey@0"
I20260924 03:09:12.513679 3042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 03:09:12.517489 3057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:12.519078 3042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:09:12.519248 3042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "0d9642458c724bbda81925f999e163ca"
format_stamp: "Formatted at 2026-09-24 03:09:12 on dist-test-slave-5l98"
server_key: "571c4ae5a4dfba8a6cde34d347173fc4"
server_key_iv: "a0a750bbd5282bed77bb724b56d36dcd"
server_key_version: "encryptionkey@0"
I20260924 03:09:12.519445 3042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:12.535728 3042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:12.540668 3042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:12.540972 3042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:12.542382 3042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:12.543736 3042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:12.543820 3042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:12.543905 3042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:12.543959 3042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:12.581073 3042 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36469
I20260924 03:09:12.581163 3169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36469 every 8 connection(s)
I20260924 03:09:12.582373 3042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:09:12.588464 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 3042
I20260924 03:09:12.588840 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:09:12.589289 388 external_mini_cluster.cc:1428] Setting key 7d3660cf8ef590a046f41ef96d3d15ee
Sep 24 03:09:12 dist-test-slave-5l98 krb5kdc[2802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:12.605370 3172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:12.585269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44191 (local address 127.0.97.62:34947)
0924 03:09:12.585707 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:12.585712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:12.586900 (+ 1188us) server_negotiation.cc:407] Connection header received
0924 03:09:12.588958 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:12.588964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:12.589056 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:12.589174 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:12.591142 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:12.592226 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:12.594011 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:12.594775 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:12.594922 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:12.599043 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:12.599048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:12.599051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:12.599082 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:12.601915 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:12.602693 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:12.602698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:12.602703 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:12.602861 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:12.603452 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:12.603455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:12.603457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:12.603720 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:12.603830 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:12.604911 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0924 03:09:12.605099 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":220,"threads_started":1}
I20260924 03:09:12.607052 3173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:12.585876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34947 (local address 127.0.97.2:44191)
0924 03:09:12.586610 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:12.586649 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:12.588634 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:12.589463 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:12.589475 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:12.590013 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:12.590805 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:12.590830 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:12.592414 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:12.592418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:12.593862 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:12.593874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:12.595006 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:12.595011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:12.595224 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:12.595969 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:12.595992 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:12.598782 (+ 2790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:12.602082 (+ 3300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:12.602092 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:12.602111 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:12.602561 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:12.603015 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:12.603025 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:12.603032 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:12.603307 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:12.603866 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:12.603878 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:12.604403 (+ 525us) client_negotiation.cc:769] Sending connection context
0924 03:09:12.604796 (+ 393us) client_negotiation.cc:240] Negotiation successful
0924 03:09:12.605409 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":283,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 03:09:12.608708 3170 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34947
I20260924 03:09:12.609092 3170 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:12.609934 3170 heartbeater.cc:511] Master 127.0.97.62:34947 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 03:09:12.611411 2847 ts_manager.cc:194] Registered new tserver with Master: 0d9642458c724bbda81925f999e163ca (127.0.97.2:36469)
I20260924 03:09:12.612241 2847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 03:09:12.638244 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34947
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:12.970312 3176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:12.970598 3176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:12.970765 3176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:12.978156 3176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:12.978274 3176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:12.978457 3176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:09:12.988600 3176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:12.989944 3176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:12.991268 3176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:13.005065 3181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:13.005172 3184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:13.005043 3182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:13.006333 3176 server_base.cc:1034] running on GCE node
I20260924 03:09:13.007077 3176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:13.008117 3176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:13.009384 3176 hybrid_clock.cc:648] HybridClock initialized: now 1790219353009274 us; error 103 us; skew 500 ppm
I20260924 03:09:13.009820 3176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:13 dist-test-slave-5l98 krb5kdc[2802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:13.013877 3176 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:09:13.015645 3176 webserver.cc:460] Webserver started at http://127.0.97.3:32787/ using document root <none> and password file <none>
I20260924 03:09:13.016863 3176 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:13.017021 3176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:13.017376 3176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:13.020385 3176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "64ed854f4d5a4e3e8c5202915026bd7c"
format_stamp: "Formatted at 2026-09-24 03:09:13 on dist-test-slave-5l98"
server_key: "4810d75e60f586f74c68e5b09dd24bb5"
server_key_iv: "d5796fc9cbc66a133f82eccccfbb6580"
server_key_version: "encryptionkey@0"
I20260924 03:09:13.021296 3176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "64ed854f4d5a4e3e8c5202915026bd7c"
format_stamp: "Formatted at 2026-09-24 03:09:13 on dist-test-slave-5l98"
server_key: "4810d75e60f586f74c68e5b09dd24bb5"
server_key_iv: "d5796fc9cbc66a133f82eccccfbb6580"
server_key_version: "encryptionkey@0"
I20260924 03:09:13.028067 3176 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260924 03:09:13.031844 3191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:13.033299 3176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 03:09:13.033497 3176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "64ed854f4d5a4e3e8c5202915026bd7c"
format_stamp: "Formatted at 2026-09-24 03:09:13 on dist-test-slave-5l98"
server_key: "4810d75e60f586f74c68e5b09dd24bb5"
server_key_iv: "d5796fc9cbc66a133f82eccccfbb6580"
server_key_version: "encryptionkey@0"
I20260924 03:09:13.033721 3176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:13.089115 3176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:13.093709 3176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:13.094026 3176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:13.095067 3176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:13.095389 3036 heartbeater.cc:503] Master 127.0.97.62:34947 was elected leader, sending a full tablet report...
I20260924 03:09:13.096837 3176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:13.096953 3176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:13.097080 3176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:13.097149 3176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:13.133015 3176 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38495
I20260924 03:09:13.133105 3303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38495 every 8 connection(s)
I20260924 03:09:13.134317 3176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:09:13.139320 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 3176
I20260924 03:09:13.139544 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:09:13.140147 388 external_mini_cluster.cc:1428] Setting key 623afd744adfacdd6642cf9ab7f8619f
Sep 24 03:09:13 dist-test-slave-5l98 krb5kdc[2802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:13.157699 3306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:13.137128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34685 (local address 127.0.97.62:34947)
0924 03:09:13.137588 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:13.137594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:13.139020 (+ 1426us) server_negotiation.cc:407] Connection header received
0924 03:09:13.141722 (+ 2702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:13.141726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:13.141791 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:13.141877 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:13.144113 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:13.145068 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:13.146678 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:13.147076 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:13.147172 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:13.151229 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:13.151234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:13.151238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:13.151274 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:13.154176 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:13.155122 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:13.155128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:13.155131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:13.155269 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:13.155747 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:13.155751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:13.155753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:13.156252 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:13.156402 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:13.157253 (+ 851us) server_negotiation.cc:299] Negotiation successful
0924 03:09:13.157422 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":259,"threads_started":1}
I20260924 03:09:13.158710 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:13.137830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34947 (local address 127.0.97.3:34685)
0924 03:09:13.138781 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:13.138840 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:13.141426 (+ 2586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:13.142211 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:13.142220 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:13.142996 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:13.143934 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:13.143977 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:13.145222 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:13.145227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:13.146539 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:13.146550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:13.147303 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:13.147308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:13.147574 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:13.148458 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:13.148482 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:13.150996 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:13.154400 (+ 3404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:13.154417 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:13.154434 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:13.154959 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:13.155386 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:13.155392 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:13.155401 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:13.155602 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:13.156390 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:13.156419 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:13.156829 (+ 410us) client_negotiation.cc:769] Sending connection context
0924 03:09:13.157146 (+ 317us) client_negotiation.cc:240] Negotiation successful
0924 03:09:13.157565 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":416,"threads_started":1}
I20260924 03:09:13.160429 3304 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34947
I20260924 03:09:13.160817 3304 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:13.161547 3304 heartbeater.cc:511] Master 127.0.97.62:34947 requested a full tablet report, sending...
I20260924 03:09:13.163003 2847 ts_manager.cc:194] Registered new tserver with Master: 64ed854f4d5a4e3e8c5202915026bd7c (127.0.97.3:38495)
I20260924 03:09:13.163806 2847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34685
I20260924 03:09:13.169353 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 03:09:13.200716 3306 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:13.198128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.0.97.62:34947)
0924 03:09:13.198407 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:13.198414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:13.198744 (+ 330us) server_negotiation.cc:407] Connection header received
0924 03:09:13.198867 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:13.198871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:13.198943 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:13.199048 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:13.200366 (+ 1318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44010: BlockingRecv error: recv got EOF from 127.0.0.1:44010 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 03:09:13.204231 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 2908
W20260924 03:09:13.207535 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 2908: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:42003: server requires authentication, but client does not have Kerberos credentials available
I20260924 03:09:13.207755 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 2908
I20260924 03:09:13.223727 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 3042
W20260924 03:09:13.226538 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 3042: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:36469: server requires authentication, but client does not have Kerberos credentials available
I20260924 03:09:13.226727 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 3042
I20260924 03:09:13.243438 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 3176
W20260924 03:09:13.247072 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 3176: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:38495: server requires authentication, but client does not have Kerberos credentials available
I20260924 03:09:13.247212 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 3176
I20260924 03:09:13.259303 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 2815
I20260924 03:09:13.260728 2877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:13.365541 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 2815
2026-09-24T03:09:13Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3791 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:09:13 dist-test-slave-5l98 krb5kdc[3322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:09:13 dist-test-slave-5l98 krb5kdc[3322](info): set up 2 sockets
Sep 24 03:09:13 dist-test-slave-5l98 krb5kdc[3322](info): commencing operation
krb5kdc: starting...
W20260924 03:09:14.473143 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:09:14 dist-test-slave-5l98 krb5kdc[3322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:09:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:09:14.681476 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44817
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:15.018428 3335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:15.018818 3335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:15.018992 3335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:15.026347 3335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:15.026536 3335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:15.026643 3335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:15.026748 3335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:15.026835 3335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:09:15.038172 3335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:15.039585 3335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:15.041107 3335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:15.053928 3341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:15.053979 3343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:15.053887 3340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:15.055348 3335 server_base.cc:1034] running on GCE node
I20260924 03:09:15.056403 3335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:15.058027 3335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:15.059259 3335 hybrid_clock.cc:648] HybridClock initialized: now 1790219355059208 us; error 36 us; skew 500 ppm
I20260924 03:09:15.059682 3335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:15 dist-test-slave-5l98 krb5kdc[3322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:15.071174 3335 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:09:15.072952 3335 webserver.cc:460] Webserver started at http://127.0.97.62:37723/ using document root <none> and password file <none>
I20260924 03:09:15.073642 3335 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:15.073743 3335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:15.073990 3335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:15.077316 3335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "b9271d589cf44b39a154016285d0f24b"
format_stamp: "Formatted at 2026-09-24 03:09:15 on dist-test-slave-5l98"
server_key: "d7a996e8705d1857d72e1c85ae335d4e"
server_key_iv: "a8259706e5558deba8db0832508fb4af"
server_key_version: "encryptionkey@0"
I20260924 03:09:15.078132 3335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "b9271d589cf44b39a154016285d0f24b"
format_stamp: "Formatted at 2026-09-24 03:09:15 on dist-test-slave-5l98"
server_key: "d7a996e8705d1857d72e1c85ae335d4e"
server_key_iv: "a8259706e5558deba8db0832508fb4af"
server_key_version: "encryptionkey@0"
I20260924 03:09:15.083925 3335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:09:15.087756 3350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:15.089489 3335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 03:09:15.089643 3335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "b9271d589cf44b39a154016285d0f24b"
format_stamp: "Formatted at 2026-09-24 03:09:15 on dist-test-slave-5l98"
server_key: "d7a996e8705d1857d72e1c85ae335d4e"
server_key_iv: "a8259706e5558deba8db0832508fb4af"
server_key_version: "encryptionkey@0"
I20260924 03:09:15.089826 3335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:15.113278 3335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:15.117780 3335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:15.118027 3335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:15.137512 3335 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44817
I20260924 03:09:15.137544 3401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44817 every 8 connection(s)
I20260924 03:09:15.138865 3335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:09:15.142345 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 3335
I20260924 03:09:15.142552 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:09:15.143111 3402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:15.143362 388 external_mini_cluster.cc:1428] Setting key fd83bcc25a77327dfd0436af84197764
I20260924 03:09:15.156080 3402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b: Bootstrap starting.
Sep 24 03:09:15 dist-test-slave-5l98 krb5kdc[3322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:15.159379 3402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:15.160656 3402 log.cc:824] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:15.163548 3402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b: No bootstrap required, opened a new log
I20260924 03:09:15.165705 3405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:15.145304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37954 (local address 127.0.97.62:44817)
0924 03:09:15.146140 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:15.146152 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:15.146193 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:09:15.148104 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:15.148127 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:15.148628 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:15.149078 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:15.150560 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:15.151892 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:15.153194 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:15.153934 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:15.154151 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:15.158302 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:15.158310 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:15.158325 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:15.158365 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:15.161673 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:15.162455 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:15.162461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:15.162467 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:15.162655 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:15.163038 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:15.163042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:15.163044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:15.163554 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:15.163842 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:15.164294 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 03:09:15.164730 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":323,"threads_started":1}
I20260924 03:09:15.169570 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:09:15.171698 3402 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9271d589cf44b39a154016285d0f24b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44817 } }
I20260924 03:09:15.172123 3402 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:15.172231 3402 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9271d589cf44b39a154016285d0f24b, State: Initialized, Role: FOLLOWER
I20260924 03:09:15.172856 3402 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9271d589cf44b39a154016285d0f24b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44817 } }
I20260924 03:09:15.173110 3402 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:09:15.173247 3402 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:09:15.173431 3402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:15.175963 3402 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9271d589cf44b39a154016285d0f24b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44817 } }
I20260924 03:09:15.176625 3402 leader_election.cc:302] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9271d589cf44b39a154016285d0f24b; no voters:
I20260924 03:09:15.177271 3402 leader_election.cc:288] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:09:15.177402 3408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:15.178305 3408 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b [term 1 LEADER]: Becoming Leader. State: Replica: b9271d589cf44b39a154016285d0f24b, State: Running, Role: LEADER
I20260924 03:09:15.178813 3408 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9271d589cf44b39a154016285d0f24b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44817 } }
I20260924 03:09:15.179616 3402 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:09:15.181440 3410 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9271d589cf44b39a154016285d0f24b. Latest consensus state: current_term: 1 leader_uuid: "b9271d589cf44b39a154016285d0f24b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9271d589cf44b39a154016285d0f24b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44817 } } }
I20260924 03:09:15.181775 3410 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:15.182339 3409 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9271d589cf44b39a154016285d0f24b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9271d589cf44b39a154016285d0f24b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44817 } } }
I20260924 03:09:15.182641 3409 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:15.184332 3413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:09:15.189607 3413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:09:15.206770 3413 catalog_manager.cc:1347] Generated new cluster ID: 3888173ba0404e57ace675a02bca01c0
I20260924 03:09:15.207067 3413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:09:15.222599 3413 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:09:15.225232 3413 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:09:15.250913 3405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:15.241507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37960 (local address 127.0.97.62:44817)
0924 03:09:15.241683 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:15.241689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:15.242263 (+ 574us) server_negotiation.cc:407] Connection header received
0924 03:09:15.242450 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:15.242456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:15.242542 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:15.242655 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:15.243850 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:15.245082 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:15.246917 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:15.247607 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:15.247698 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:15.249181 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:15.249187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:15.249192 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:15.250066 (+ 874us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:15.250120 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:15.250161 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:15.250282 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:15.250451 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 03:09:15.250561 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 03:09:15.252635 3413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9271d589cf44b39a154016285d0f24b: Generated new TSK 0
I20260924 03:09:15.253544 3413 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:09:15.265511 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 3335
I20260924 03:09:15.266646 3397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:15.358922 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 3335
2026-09-24T03:09:15Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1990 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:09:15 dist-test-slave-5l98 krb5kdc[3436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:09:15 dist-test-slave-5l98 krb5kdc[3436](info): set up 2 sockets
Sep 24 03:09:15 dist-test-slave-5l98 krb5kdc[3436](info): commencing operation
krb5kdc: starting...
W20260924 03:09:16.521149 388 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:09:16 dist-test-slave-5l98 krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:09:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:09:16.722628 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42455
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:17.063973 3449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:17.064369 3449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:17.064625 3449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:17.072137 3449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:17.072299 3449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:17.072409 3449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:17.072544 3449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:17.072633 3449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:09:17.082640 3449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:17.084052 3449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:17.085448 3449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:17.097865 3454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:17.097954 3455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:17.097932 3457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:17.099167 3449 server_base.cc:1034] running on GCE node
I20260924 03:09:17.100055 3449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:17.101543 3449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:17.102807 3449 hybrid_clock.cc:648] HybridClock initialized: now 1790219357102759 us; error 34 us; skew 500 ppm
I20260924 03:09:17.103190 3449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:17 dist-test-slave-5l98 krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:17.106683 3449 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:09:17.108314 3449 webserver.cc:460] Webserver started at http://127.0.97.62:40809/ using document root <none> and password file <none>
I20260924 03:09:17.109102 3449 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:17.109248 3449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:17.109557 3449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:17.112571 3449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "2cfdf744a7c64b8ea2f9e60ff20f547b"
format_stamp: "Formatted at 2026-09-24 03:09:17 on dist-test-slave-5l98"
server_key: "358be004597c51bb0f8e24951d155913"
server_key_iv: "3d03b6b4c3d9f12f890b734f5ccdde94"
server_key_version: "encryptionkey@0"
I20260924 03:09:17.113353 3449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "2cfdf744a7c64b8ea2f9e60ff20f547b"
format_stamp: "Formatted at 2026-09-24 03:09:17 on dist-test-slave-5l98"
server_key: "358be004597c51bb0f8e24951d155913"
server_key_iv: "3d03b6b4c3d9f12f890b734f5ccdde94"
server_key_version: "encryptionkey@0"
I20260924 03:09:17.118698 3449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 03:09:17.122608 3464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:17.124092 3449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:09:17.124316 3449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "2cfdf744a7c64b8ea2f9e60ff20f547b"
format_stamp: "Formatted at 2026-09-24 03:09:17 on dist-test-slave-5l98"
server_key: "358be004597c51bb0f8e24951d155913"
server_key_iv: "3d03b6b4c3d9f12f890b734f5ccdde94"
server_key_version: "encryptionkey@0"
I20260924 03:09:17.124619 3449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:17.153841 3449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:17.160091 3449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:17.160470 3449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:17.182204 3449 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42455
I20260924 03:09:17.182305 3515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42455 every 8 connection(s)
I20260924 03:09:17.183796 3449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:09:17.187772 3516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:17.193392 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 3449
I20260924 03:09:17.193605 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:09:17.194464 388 external_mini_cluster.cc:1428] Setting key 1fa1ca2e73567b9125a40ebf373f7339
I20260924 03:09:17.199971 3516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b: Bootstrap starting.
I20260924 03:09:17.203773 3516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:17.205073 3516 log.cc:824] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:17.209031 3516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b: No bootstrap required, opened a new log
Sep 24 03:09:17 dist-test-slave-5l98 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:17.219862 3516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cfdf744a7c64b8ea2f9e60ff20f547b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42455 } }
I20260924 03:09:17.220321 3516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:17.220433 3516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cfdf744a7c64b8ea2f9e60ff20f547b, State: Initialized, Role: FOLLOWER
I20260924 03:09:17.221100 3519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:17.197045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47652 (local address 127.0.97.62:42455)
0924 03:09:17.197955 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:17.197973 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:17.198017 (+ 44us) server_negotiation.cc:407] Connection header received
0924 03:09:17.199650 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:17.199673 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:17.200138 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:17.200477 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:17.202228 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:17.203691 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:17.205483 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:17.206666 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:17.206827 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:17.211721 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:17.211734 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:17.211762 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:17.211809 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:17.216045 (+ 4236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:17.216981 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:17.216991 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:17.217000 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:17.217226 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:17.217718 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:17.217723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:17.217727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:17.218502 (+ 775us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:17.218816 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:17.219216 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 03:09:17.219669 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":445,"threads_started":1}
I20260924 03:09:17.221235 3516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cfdf744a7c64b8ea2f9e60ff20f547b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42455 } }
I20260924 03:09:17.221524 3516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:09:17.221689 3516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:09:17.221928 3516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:17.225003 3516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cfdf744a7c64b8ea2f9e60ff20f547b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42455 } }
I20260924 03:09:17.225505 3516 leader_election.cc:302] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cfdf744a7c64b8ea2f9e60ff20f547b; no voters:
I20260924 03:09:17.226154 3516 leader_election.cc:288] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:09:17.226336 3521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:17.227658 3521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b [term 1 LEADER]: Becoming Leader. State: Replica: 2cfdf744a7c64b8ea2f9e60ff20f547b, State: Running, Role: LEADER
I20260924 03:09:17.228374 3516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:09:17.228297 3521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cfdf744a7c64b8ea2f9e60ff20f547b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42455 } }
I20260924 03:09:17.231292 3522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cfdf744a7c64b8ea2f9e60ff20f547b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cfdf744a7c64b8ea2f9e60ff20f547b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42455 } } }
I20260924 03:09:17.231647 3522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:17.232453 3523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cfdf744a7c64b8ea2f9e60ff20f547b. Latest consensus state: current_term: 1 leader_uuid: "2cfdf744a7c64b8ea2f9e60ff20f547b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cfdf744a7c64b8ea2f9e60ff20f547b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42455 } } }
I20260924 03:09:17.232895 3523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:17.234824 3529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:09:17.240511 3529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:09:17.250669 3529 catalog_manager.cc:1347] Generated new cluster ID: ce4acefba6d9402d80411f9ef249d687
I20260924 03:09:17.250818 3529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:09:17.273118 3529 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:09:17.274523 3529 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:09:17.280642 3529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cfdf744a7c64b8ea2f9e60ff20f547b: Generated new TSK 0
I20260924 03:09:17.281643 3529 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 03:09:17.346359 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42455
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:17.706522 3542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:17.706897 3542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:17.707052 3542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:17.714564 3542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:17.714684 3542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:17.714879 3542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:09:17.724695 3542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:17.726039 3542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:17.727389 3542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:17.741797 3548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:17.741739 3547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:17.741775 3550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:17.742429 3542 server_base.cc:1034] running on GCE node
I20260924 03:09:17.743043 3542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:17.743939 3542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:17.745151 3542 hybrid_clock.cc:648] HybridClock initialized: now 1790219357745095 us; error 42 us; skew 500 ppm
I20260924 03:09:17.745487 3542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:17 dist-test-slave-5l98 krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:17.748885 3542 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:09:17.750367 3542 webserver.cc:460] Webserver started at http://127.0.97.1:33835/ using document root <none> and password file <none>
I20260924 03:09:17.751081 3542 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:17.751225 3542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:17.751538 3542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:17.754513 3542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "a68e8f6fea4f47e8be4bc5c107b7c18e"
format_stamp: "Formatted at 2026-09-24 03:09:17 on dist-test-slave-5l98"
server_key: "38c2f33a9da4279e1ff90da6c9778262"
server_key_iv: "d5659b78ad2a6e7306a897f59288c405"
server_key_version: "encryptionkey@0"
I20260924 03:09:17.755302 3542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a68e8f6fea4f47e8be4bc5c107b7c18e"
format_stamp: "Formatted at 2026-09-24 03:09:17 on dist-test-slave-5l98"
server_key: "38c2f33a9da4279e1ff90da6c9778262"
server_key_iv: "d5659b78ad2a6e7306a897f59288c405"
server_key_version: "encryptionkey@0"
I20260924 03:09:17.760814 3542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:09:17.764647 3557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:17.766119 3542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:09:17.766321 3542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "a68e8f6fea4f47e8be4bc5c107b7c18e"
format_stamp: "Formatted at 2026-09-24 03:09:17 on dist-test-slave-5l98"
server_key: "38c2f33a9da4279e1ff90da6c9778262"
server_key_iv: "d5659b78ad2a6e7306a897f59288c405"
server_key_version: "encryptionkey@0"
I20260924 03:09:17.766548 3542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:17.797073 3542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:17.801414 3542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:17.801740 3542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:17.802719 3542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:17.804106 3542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:17.804216 3542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:17.804378 3542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:17.804464 3542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:17.841130 3542 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40745
I20260924 03:09:17.841173 3669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40745 every 8 connection(s)
I20260924 03:09:17.842564 3542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:09:17.849563 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 3542
I20260924 03:09:17.849768 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:09:17.850373 388 external_mini_cluster.cc:1428] Setting key 12e8d910b78e0db435d3278ce35da848
Sep 24 03:09:17 dist-test-slave-5l98 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:17.864604 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:17.845353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43389 (local address 127.0.97.62:42455)
0924 03:09:17.845952 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:17.845957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:17.847002 (+ 1045us) server_negotiation.cc:407] Connection header received
0924 03:09:17.849043 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:17.849049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:17.849156 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:17.849283 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:17.851030 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:17.851959 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:17.853959 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:17.854414 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:17.854569 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:17.858711 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:17.858716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:17.858719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:17.858754 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:17.861465 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:17.862260 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:17.862264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:17.862266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:17.862384 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:17.862833 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:17.862837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:17.862839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:17.863086 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:17.863254 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:17.863972 (+ 718us) server_negotiation.cc:299] Negotiation successful
0924 03:09:17.864201 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":272,"threads_started":1}
I20260924 03:09:17.865674 3673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:17.845840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42455 (local address 127.0.97.1:43389)
0924 03:09:17.846647 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:17.846694 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:17.848757 (+ 2063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:17.849525 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:17.849534 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:17.850162 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:17.850806 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:17.850824 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:17.852133 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:17.852139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:17.853761 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:17.853778 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:17.854663 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:17.854673 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:17.854932 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:17.855662 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:17.855685 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:17.858429 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:17.861645 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:17.861655 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:17.861674 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:17.862112 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:17.862497 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:17.862502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:17.862506 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:17.862695 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:17.863216 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:17.863224 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:17.863582 (+ 358us) client_negotiation.cc:769] Sending connection context
0924 03:09:17.863860 (+ 278us) client_negotiation.cc:240] Negotiation successful
0924 03:09:17.864421 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":307,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 03:09:17.867569 3670 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42455
I20260924 03:09:17.867967 3670 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:17.868798 3670 heartbeater.cc:511] Master 127.0.97.62:42455 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 03:09:17.871369 3481 ts_manager.cc:194] Registered new tserver with Master: a68e8f6fea4f47e8be4bc5c107b7c18e (127.0.97.1:40745)
I20260924 03:09:17.872915 3481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 03:09:17.898429 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42455
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:18.231393 3676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:18.231699 3676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:18.231860 3676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:18.239611 3676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:18.239733 3676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:18.239918 3676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:09:18.250290 3676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:18.251619 3676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:18.253015 3676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:18.266346 3684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:18.266386 3681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:18.266317 3682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:18.267696 3676 server_base.cc:1034] running on GCE node
I20260924 03:09:18.268308 3676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:18.269420 3676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:18.270658 3676 hybrid_clock.cc:648] HybridClock initialized: now 1790219358270575 us; error 104 us; skew 500 ppm
I20260924 03:09:18.270994 3676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:18 dist-test-slave-5l98 krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:18.274554 3676 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:09:18.276225 3676 webserver.cc:460] Webserver started at http://127.0.97.2:33899/ using document root <none> and password file <none>
I20260924 03:09:18.277020 3676 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:18.277163 3676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:18.277468 3676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:18.280344 3676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "917c3609a76148a5b046b675e471ef52"
format_stamp: "Formatted at 2026-09-24 03:09:18 on dist-test-slave-5l98"
server_key: "db8430fc84755dbb666cfb104b4f14bf"
server_key_iv: "4024a3b0a2bf66eb947478c111e6ad79"
server_key_version: "encryptionkey@0"
I20260924 03:09:18.281265 3676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "917c3609a76148a5b046b675e471ef52"
format_stamp: "Formatted at 2026-09-24 03:09:18 on dist-test-slave-5l98"
server_key: "db8430fc84755dbb666cfb104b4f14bf"
server_key_iv: "4024a3b0a2bf66eb947478c111e6ad79"
server_key_version: "encryptionkey@0"
I20260924 03:09:18.286474 3676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:09:18.290079 3691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:18.291626 3676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:09:18.291805 3676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "917c3609a76148a5b046b675e471ef52"
format_stamp: "Formatted at 2026-09-24 03:09:18 on dist-test-slave-5l98"
server_key: "db8430fc84755dbb666cfb104b4f14bf"
server_key_iv: "4024a3b0a2bf66eb947478c111e6ad79"
server_key_version: "encryptionkey@0"
I20260924 03:09:18.292006 3676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:18.304327 3676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:18.308771 3676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:18.309151 3676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:18.310181 3676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:18.311539 3676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:18.311642 3676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:18.311795 3676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:18.311875 3676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:18.346011 3676 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45467
I20260924 03:09:18.346109 3803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45467 every 8 connection(s)
I20260924 03:09:18.347312 3676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:09:18.347886 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 3676
I20260924 03:09:18.348083 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:09:18.348763 388 external_mini_cluster.cc:1428] Setting key f1ae1ad6ae5f77914c46d13a61653e95
Sep 24 03:09:18 dist-test-slave-5l98 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 03:09:18.372982 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:18.350072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48449 (local address 127.0.97.62:42455)
0924 03:09:18.350267 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:18.350275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:18.351966 (+ 1691us) server_negotiation.cc:407] Connection header received
0924 03:09:18.355061 (+ 3095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:18.355066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:18.355138 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:18.355279 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:18.357550 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.358733 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.360416 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.361202 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.361347 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:18.365375 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:18.365385 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:18.365401 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:18.365450 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:18.369170 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:18.370123 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:18.370127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:18.370130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:18.370260 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:18.370889 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:18.370967 (+ 78us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:18.370969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:18.371231 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:18.371351 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:18.372248 (+ 897us) server_negotiation.cc:299] Negotiation successful
0924 03:09:18.372437 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:09:18.374306 3806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:18.350783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42455 (local address 127.0.97.2:48449)
0924 03:09:18.351766 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:18.351823 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:18.354699 (+ 2876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:18.355553 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:18.355569 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:18.356397 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:18.357335 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:18.357359 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.358900 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.358904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:18.360235 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:18.360255 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.361455 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.361461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:18.361707 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:18.362452 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:18.362483 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:18.365121 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:18.369388 (+ 4267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:18.369396 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:18.369411 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:18.369956 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:18.370410 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:18.370414 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:18.370419 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:18.370748 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:18.371398 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:18.371406 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:18.371891 (+ 485us) client_negotiation.cc:769] Sending connection context
0924 03:09:18.372134 (+ 243us) client_negotiation.cc:240] Negotiation successful
0924 03:09:18.372721 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":429,"threads_started":1}
I20260924 03:09:18.375942 3804 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42455
I20260924 03:09:18.376291 3804 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:18.377228 3804 heartbeater.cc:511] Master 127.0.97.62:42455 requested a full tablet report, sending...
I20260924 03:09:18.379036 3481 ts_manager.cc:194] Registered new tserver with Master: 917c3609a76148a5b046b675e471ef52 (127.0.97.2:45467)
I20260924 03:09:18.380332 3481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 03:09:18.401453 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42455
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:18.729787 3809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:18.730155 3809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:18.730259 3809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:18.738514 3809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:18.738631 3809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:18.738755 3809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:09:18.750419 3809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:18.751837 3809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:18.753314 3809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:18.767896 3814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:18.767931 3815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:18.767863 3817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:18.769310 3809 server_base.cc:1034] running on GCE node
I20260924 03:09:18.770049 3809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:18.771136 3809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:18.772344 3809 hybrid_clock.cc:648] HybridClock initialized: now 1790219358772274 us; error 60 us; skew 500 ppm
I20260924 03:09:18.772696 3809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:18 dist-test-slave-5l98 krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:18.775982 3809 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:09:18.777555 3809 webserver.cc:460] Webserver started at http://127.0.97.3:35995/ using document root <none> and password file <none>
I20260924 03:09:18.778218 3809 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:18.778326 3809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:18.778553 3809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:18.781415 3809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "08e78bf80c9342339b042e3220dbd762"
format_stamp: "Formatted at 2026-09-24 03:09:18 on dist-test-slave-5l98"
server_key: "7b43d8b88c72e832dd410b3747299e2d"
server_key_iv: "762e8fc70d1fcd89783732445f0e3052"
server_key_version: "encryptionkey@0"
I20260924 03:09:18.782130 3809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "08e78bf80c9342339b042e3220dbd762"
format_stamp: "Formatted at 2026-09-24 03:09:18 on dist-test-slave-5l98"
server_key: "7b43d8b88c72e832dd410b3747299e2d"
server_key_iv: "762e8fc70d1fcd89783732445f0e3052"
server_key_version: "encryptionkey@0"
I20260924 03:09:18.787104 3809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:09:18.790647 3824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:18.792207 3809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:09:18.792423 3809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "08e78bf80c9342339b042e3220dbd762"
format_stamp: "Formatted at 2026-09-24 03:09:18 on dist-test-slave-5l98"
server_key: "7b43d8b88c72e832dd410b3747299e2d"
server_key_iv: "762e8fc70d1fcd89783732445f0e3052"
server_key_version: "encryptionkey@0"
I20260924 03:09:18.792725 3809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:18.810664 3809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:18.815040 3809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:18.815449 3809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:18.816403 3809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:18.817762 3809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:18.817878 3809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:18.818033 3809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:18.818104 3809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:18.855363 3809 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38119
I20260924 03:09:18.855468 3936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38119 every 8 connection(s)
I20260924 03:09:18.857437 3809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:09:18.861064 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 3809
I20260924 03:09:18.861249 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:09:18.861855 388 external_mini_cluster.cc:1428] Setting key 5169f292a658c218f76b211d6d03b407
Sep 24 03:09:18 dist-test-slave-5l98 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:18.876001 3670 heartbeater.cc:503] Master 127.0.97.62:42455 was elected leader, sending a full tablet report...
I20260924 03:09:18.881362 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:18.863447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45275 (local address 127.0.97.62:42455)
0924 03:09:18.864046 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:18.864050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:18.864094 (+ 44us) server_negotiation.cc:407] Connection header received
0924 03:09:18.865595 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:18.865601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:18.865745 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:18.865893 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:18.867520 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.868317 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.870074 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.870464 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.870567 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:18.874811 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:18.874816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:18.874819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:18.874855 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:18.878190 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:18.878958 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:18.878963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:18.878967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:18.879100 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:18.879561 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:18.879573 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:18.879575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:18.879842 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:18.879998 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:18.880795 (+ 797us) server_negotiation.cc:299] Negotiation successful
0924 03:09:18.881014 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471}
I20260924 03:09:18.882334 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:18.861570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42455 (local address 127.0.97.3:45275)
0924 03:09:18.862613 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:18.862672 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:18.865271 (+ 2599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:18.866090 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:18.866099 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:18.866758 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:18.867347 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:18.867364 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.868472 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.868476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:18.869898 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:18.869910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.870687 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.870692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:18.870923 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:18.871841 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:18.871868 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:18.874543 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:18.878351 (+ 3808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:18.878361 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:18.878384 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:18.878820 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:18.879215 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:18.879220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:18.879224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:18.879443 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:18.879975 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:18.879983 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:18.880350 (+ 367us) client_negotiation.cc:769] Sending connection context
0924 03:09:18.880671 (+ 321us) client_negotiation.cc:240] Negotiation successful
0924 03:09:18.881212 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":420,"threads_started":1}
I20260924 03:09:18.883958 3937 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42455
I20260924 03:09:18.884305 3937 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:18.885185 3937 heartbeater.cc:511] Master 127.0.97.62:42455 requested a full tablet report, sending...
I20260924 03:09:18.886739 3481 ts_manager.cc:194] Registered new tserver with Master: 08e78bf80c9342339b042e3220dbd762 (127.0.97.3:38119)
I20260924 03:09:18.887719 3481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45275
I20260924 03:09:18.891774 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:09:18 dist-test-slave-5l98 krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219358, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 03:09:18 dist-test-slave-5l98 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219358, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:18.942821 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:18.927398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.0.97.62:42455)
0924 03:09:18.927568 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:18.927573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:18.928151 (+ 578us) server_negotiation.cc:407] Connection header received
0924 03:09:18.928341 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:18.928345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:18.928428 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:18.928614 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:18.930371 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.931120 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.932482 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.932919 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.933012 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:18.937580 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:18.937587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:18.937593 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:18.937637 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:18.940219 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:18.941068 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:18.941072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:18.941075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:18.941193 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:18.941613 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:18.941616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:18.941618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:18.941928 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:18.942108 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:18.942412 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 03:09:18.942576 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 03:09:18.943502 3481 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47664
I20260924 03:09:18.945340 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 3542
Sep 24 03:09:18 dist-test-slave-5l98 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219358, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:09:18.962576 3948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:18.946537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53810 (local address 127.0.97.1:40745)
0924 03:09:18.947247 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:18.947255 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:18.947281 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:09:18.947342 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:18.947349 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:18.947580 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:18.947710 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:18.948931 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.950015 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.951215 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.951848 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.951986 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:18.956293 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:18.956304 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:18.956321 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:18.956359 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:18.959415 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:18.960218 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:18.960231 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:18.960234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:18.960377 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:18.960884 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:18.960888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:18.960890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:18.961246 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:18.961463 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:18.961857 (+ 394us) server_negotiation.cc:299] Negotiation successful
0924 03:09:18.962085 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":369,"threads_started":1}
W20260924 03:09:18.964141 3665 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53810
W20260924 03:09:18.965495 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 3542: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:09:18.965752 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 3542
I20260924 03:09:18.981207 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 3676
Sep 24 03:09:18 dist-test-slave-5l98 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219358, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 03:09:18.998251 3949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:18.982310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.0.97.2:45467)
0924 03:09:18.982838 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:18.982846 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:18.982870 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:09:18.983011 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:18.983017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:18.983236 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:18.983413 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:18.984799 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.985795 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.987031 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.987609 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.987783 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:18.992112 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:18.992126 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:18.992142 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:18.992179 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:18.995151 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:18.995847 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:18.995855 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:18.995857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:18.996044 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:18.996442 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:18.996448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:18.996451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:18.996937 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:18.997281 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:18.997626 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 03:09:18.997888 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":281,"threads_started":1}
W20260924 03:09:18.999570 3799 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42876
W20260924 03:09:19.000931 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 3676: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:09:19.001091 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 3676
I20260924 03:09:19.016528 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 3809
Sep 24 03:09:19 dist-test-slave-5l98 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219358, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 03:09:19.033658 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:19.017622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.0.97.3:38119)
0924 03:09:19.018226 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:19.018233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:19.018264 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:09:19.018324 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:19.018331 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:19.018570 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:19.018761 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:19.020238 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:19.021423 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:19.022824 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:19.023447 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:19.023607 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:19.028098 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:19.028109 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:19.028124 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:19.028157 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:19.031053 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:19.031718 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:19.031725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:19.031727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:19.031835 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:19.032163 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:19.032166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:19.032168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:19.032534 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:19.032774 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:19.033195 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 03:09:19.033384 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":302,"threads_started":1}
W20260924 03:09:19.035214 3932 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40370
W20260924 03:09:19.036401 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 3809: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:09:19.036664 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 3809
I20260924 03:09:19.051955 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 3449
I20260924 03:09:19.053231 3511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:19.158336 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 3449
2026-09-24T03:09:19Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3802 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:09:19 dist-test-slave-5l98 krb5kdc[3954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:09:19 dist-test-slave-5l98 krb5kdc[3954](info): set up 2 sockets
Sep 24 03:09:19 dist-test-slave-5l98 krb5kdc[3954](info): commencing operation
krb5kdc: starting...
W20260924 03:09:20.269130 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.007s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:09:20 dist-test-slave-5l98 krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:09:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:09:20.486552 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34503
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:20.815435 3967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:20.815737 3967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:20.815900 3967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:20.823267 3967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:20.823388 3967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:20.823482 3967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:20.823565 3967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:20.823625 3967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:09:20.833539 3967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:20.834978 3967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:20.836288 3967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:20.848544 3975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:20.848604 3973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:20.848579 3972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:20.849222 3967 server_base.cc:1034] running on GCE node
I20260924 03:09:20.850024 3967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:20.851619 3967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:20.852854 3967 hybrid_clock.cc:648] HybridClock initialized: now 1790219360852805 us; error 49 us; skew 500 ppm
I20260924 03:09:20.853178 3967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:20 dist-test-slave-5l98 krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:20.862341 3967 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:09:20.864001 3967 webserver.cc:460] Webserver started at http://127.0.97.62:39301/ using document root <none> and password file <none>
I20260924 03:09:20.864784 3967 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:20.864893 3967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:20.865146 3967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:20.868116 3967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "50b65609f55343588edab8fb066ff013"
format_stamp: "Formatted at 2026-09-24 03:09:20 on dist-test-slave-5l98"
server_key: "c1e07c23295e7457e1f509c6143374dd"
server_key_iv: "d40d537f1b35706bbd33c607556b0a19"
server_key_version: "encryptionkey@0"
I20260924 03:09:20.868947 3967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "50b65609f55343588edab8fb066ff013"
format_stamp: "Formatted at 2026-09-24 03:09:20 on dist-test-slave-5l98"
server_key: "c1e07c23295e7457e1f509c6143374dd"
server_key_iv: "d40d537f1b35706bbd33c607556b0a19"
server_key_version: "encryptionkey@0"
I20260924 03:09:20.874262 3967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:09:20.877821 3982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:20.879441 3967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:09:20.879601 3967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "50b65609f55343588edab8fb066ff013"
format_stamp: "Formatted at 2026-09-24 03:09:20 on dist-test-slave-5l98"
server_key: "c1e07c23295e7457e1f509c6143374dd"
server_key_iv: "d40d537f1b35706bbd33c607556b0a19"
server_key_version: "encryptionkey@0"
I20260924 03:09:20.879770 3967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:20.899432 3967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:20.904515 3967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:20.905037 3967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:20.925210 3967 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34503
I20260924 03:09:20.925328 4033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34503 every 8 connection(s)
I20260924 03:09:20.926649 3967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:09:20.931241 4034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:20.936542 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 3967
I20260924 03:09:20.936776 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:09:20.937546 388 external_mini_cluster.cc:1428] Setting key ebca560903745e7dcbdf23ec3e195ef7
I20260924 03:09:20.943274 4034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013: Bootstrap starting.
I20260924 03:09:20.947145 4034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:20.948478 4034 log.cc:824] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:20.952452 4034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013: No bootstrap required, opened a new log
Sep 24 03:09:20 dist-test-slave-5l98 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:20.963771 4034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b65609f55343588edab8fb066ff013" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34503 } }
I20260924 03:09:20.964176 4037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:20.939790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.0.97.62:34503)
0924 03:09:20.940722 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:20.940737 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:20.940783 (+ 46us) server_negotiation.cc:407] Connection header received
0924 03:09:20.942666 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:20.942691 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:20.943107 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:20.943510 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:20.945285 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:20.946641 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:20.948664 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:20.949780 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:20.949990 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:20.954804 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:20.954819 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:20.954840 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:20.954908 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:20.959138 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:20.959960 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:20.959969 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:20.959978 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:20.960195 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:20.960688 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:20.960693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:20.960697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:20.961486 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:20.961810 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:20.962297 (+ 487us) server_negotiation.cc:299] Negotiation successful
0924 03:09:20.962786 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":427,"threads_started":1}
I20260924 03:09:20.964244 4034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:20.964332 4034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50b65609f55343588edab8fb066ff013, State: Initialized, Role: FOLLOWER
I20260924 03:09:20.965081 4034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b65609f55343588edab8fb066ff013" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34503 } }
I20260924 03:09:20.965304 4034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:09:20.965466 4034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:09:20.965657 4034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:20.968245 4034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b65609f55343588edab8fb066ff013" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34503 } }
I20260924 03:09:20.968854 4034 leader_election.cc:302] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50b65609f55343588edab8fb066ff013; no voters:
I20260924 03:09:20.969444 4034 leader_election.cc:288] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:09:20.969597 4039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:20.970944 4039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013 [term 1 LEADER]: Becoming Leader. State: Replica: 50b65609f55343588edab8fb066ff013, State: Running, Role: LEADER
I20260924 03:09:20.971446 4039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b65609f55343588edab8fb066ff013" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34503 } }
I20260924 03:09:20.971688 4034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:09:20.974437 4040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50b65609f55343588edab8fb066ff013" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b65609f55343588edab8fb066ff013" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34503 } } }
I20260924 03:09:20.975147 4040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:20.974658 4041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50b65609f55343588edab8fb066ff013. Latest consensus state: current_term: 1 leader_uuid: "50b65609f55343588edab8fb066ff013" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b65609f55343588edab8fb066ff013" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34503 } } }
I20260924 03:09:20.975703 4041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:20.976262 4047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:09:20.982532 4047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:09:20.992075 4047 catalog_manager.cc:1347] Generated new cluster ID: 6797edafb0d54a7b8316606d9f6b67e0
I20260924 03:09:20.992216 4047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:09:21.010470 4047 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:09:21.011909 4047 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:09:21.027518 4047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50b65609f55343588edab8fb066ff013: Generated new TSK 0
I20260924 03:09:21.028719 4047 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 03:09:21.094087 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34503
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:21.433408 4060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:21.433799 4060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:21.433974 4060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:21.441268 4060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:21.441383 4060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:21.441565 4060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:09:21.452353 4060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:21.453758 4060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:21.455143 4060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:21.469619 4065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:21.469641 4066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:21.470263 4068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:21.470781 4060 server_base.cc:1034] running on GCE node
I20260924 03:09:21.471592 4060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:21.472762 4060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:21.474042 4060 hybrid_clock.cc:648] HybridClock initialized: now 1790219361473964 us; error 72 us; skew 500 ppm
I20260924 03:09:21.474454 4060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:21 dist-test-slave-5l98 krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:21.477959 4060 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:09:21.479421 4060 webserver.cc:460] Webserver started at http://127.0.97.1:38849/ using document root <none> and password file <none>
I20260924 03:09:21.480110 4060 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:21.480238 4060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:21.480629 4060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:21.483775 4060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "81107b423ba7487db1eabddd998174c2"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-5l98"
server_key: "fa84053f4262d672c646b4e3036025a8"
server_key_iv: "97b604b0465c056ab074b230d7ca2617"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.484805 4060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "81107b423ba7487db1eabddd998174c2"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-5l98"
server_key: "fa84053f4262d672c646b4e3036025a8"
server_key_iv: "97b604b0465c056ab074b230d7ca2617"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.490435 4060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:09:21.494439 4075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:21.496086 4060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:09:21.496289 4060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "81107b423ba7487db1eabddd998174c2"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-5l98"
server_key: "fa84053f4262d672c646b4e3036025a8"
server_key_iv: "97b604b0465c056ab074b230d7ca2617"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.496553 4060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:21.512161 4060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:21.516645 4060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:21.517000 4060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:21.518002 4060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:21.519479 4060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:21.519584 4060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:21.519743 4060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:21.519822 4060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:21.554764 4060 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41863
I20260924 03:09:21.554859 4187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41863 every 8 connection(s)
I20260924 03:09:21.556094 4060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:09:21.556555 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 4060
I20260924 03:09:21.556787 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:09:21.557610 388 external_mini_cluster.cc:1428] Setting key d0ae2f156848fc58ec6c9ec9294a0f82
Sep 24 03:09:21 dist-test-slave-5l98 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 03:09:21.579594 4190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:21.558946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43331 (local address 127.0.97.62:34503)
0924 03:09:21.559428 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:21.559432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:21.560737 (+ 1305us) server_negotiation.cc:407] Connection header received
0924 03:09:21.562538 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:21.562546 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:21.562665 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:21.562806 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:21.564592 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:21.565377 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:21.568049 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:21.568449 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:21.568631 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:21.573444 (+ 4813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:21.573450 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:21.573453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:21.573485 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:21.576082 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:21.577019 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:21.577028 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:21.577031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:21.577156 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:21.577590 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:21.577595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:21.577598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:21.578021 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:21.578176 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:21.578990 (+ 814us) server_negotiation.cc:299] Negotiation successful
0924 03:09:21.579215 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":236,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 03:09:21.580794 4191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:21.559718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34503 (local address 127.0.97.1:43331)
0924 03:09:21.560585 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:21.560630 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:21.562258 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:21.563072 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:21.563080 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:21.563591 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:21.564342 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:21.564372 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:21.565546 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:21.565554 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:21.567873 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:21.567892 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:21.568732 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:21.568741 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:21.569103 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:21.569939 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:21.569974 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:21.573199 (+ 3225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:21.576245 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:21.576254 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:21.576273 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:21.576894 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:21.577268 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:21.577273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:21.577277 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:21.577468 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:21.578153 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:21.578162 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:21.578550 (+ 388us) client_negotiation.cc:769] Sending connection context
0924 03:09:21.578873 (+ 323us) client_negotiation.cc:240] Negotiation successful
0924 03:09:21.579450 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":359,"threads_started":1}
I20260924 03:09:21.582429 4188 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34503
I20260924 03:09:21.582764 4188 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:21.583653 4188 heartbeater.cc:511] Master 127.0.97.62:34503 requested a full tablet report, sending...
I20260924 03:09:21.586045 3999 ts_manager.cc:194] Registered new tserver with Master: 81107b423ba7487db1eabddd998174c2 (127.0.97.1:41863)
I20260924 03:09:21.587466 3999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 03:09:21.604803 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34503
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:21.934899 4194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:21.935261 4194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:21.935452 4194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:21.943573 4194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:21.943748 4194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:21.943949 4194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:09:21.953925 4194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:21.955281 4194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:21.956634 4194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:21.970602 4200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:21.970641 4202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:21.971179 4199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:21.971721 4194 server_base.cc:1034] running on GCE node
I20260924 03:09:21.972424 4194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:21.973397 4194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:21.974606 4194 hybrid_clock.cc:648] HybridClock initialized: now 1790219361974534 us; error 68 us; skew 500 ppm
I20260924 03:09:21.974969 4194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:21 dist-test-slave-5l98 krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:21.978808 4194 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:09:21.980365 4194 webserver.cc:460] Webserver started at http://127.0.97.2:41365/ using document root <none> and password file <none>
I20260924 03:09:21.981119 4194 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:21.981254 4194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:21.981594 4194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:21.984972 4194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "ef71825cc92d4069b3f203f81d62cf05"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-5l98"
server_key: "db4a919bb332ed75421c447274e29a56"
server_key_iv: "728e58c5de9b74fee675e571e93ffc1d"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.985770 4194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ef71825cc92d4069b3f203f81d62cf05"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-5l98"
server_key: "db4a919bb332ed75421c447274e29a56"
server_key_iv: "728e58c5de9b74fee675e571e93ffc1d"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.991220 4194 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:09:21.995157 4209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:21.996783 4194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 03:09:21.996964 4194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "ef71825cc92d4069b3f203f81d62cf05"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-5l98"
server_key: "db4a919bb332ed75421c447274e29a56"
server_key_iv: "728e58c5de9b74fee675e571e93ffc1d"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.997176 4194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:22.012297 4194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:22.017159 4194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:22.017503 4194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:22.018569 4194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:22.019985 4194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:22.020102 4194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:22.020251 4194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:22.020326 4194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:22.056607 4194 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43259
I20260924 03:09:22.056697 4321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43259 every 8 connection(s)
I20260924 03:09:22.058068 4194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:09:22.064044 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 4194
I20260924 03:09:22.064271 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:09:22.064785 388 external_mini_cluster.cc:1428] Setting key f160bbb19918c75f68366e585ec8b07c
Sep 24 03:09:22 dist-test-slave-5l98 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:22.083005 4190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:22.060972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49911 (local address 127.0.97.62:34503)
0924 03:09:22.061189 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:22.061194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:22.062424 (+ 1230us) server_negotiation.cc:407] Connection header received
0924 03:09:22.064380 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:22.064386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:22.064483 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:22.064651 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:22.066356 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:22.067077 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:22.069774 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:22.070690 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:22.070843 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:22.075412 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:22.075418 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:22.075421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:22.075460 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:22.079196 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:22.080092 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:22.080100 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:22.080107 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:22.080279 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:22.080824 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:22.080832 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:22.080836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:22.081286 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:22.081457 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:22.082316 (+ 859us) server_negotiation.cc:299] Negotiation successful
0924 03:09:22.082557 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 03:09:22.083738 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:22.061489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34503 (local address 127.0.97.2:49911)
0924 03:09:22.062249 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:22.062289 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:22.064029 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:22.064921 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:22.064933 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:22.065479 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:22.066187 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:22.066205 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:22.067264 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:22.067268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:22.069516 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:22.069537 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:22.071017 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:22.071027 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:22.071415 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:22.072128 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:22.072151 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:22.075100 (+ 2949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:22.079409 (+ 4309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:22.079449 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:22.079465 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:22.079941 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:22.080443 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:22.080451 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:22.080455 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:22.080703 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:22.081445 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:22.081457 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:22.081904 (+ 447us) client_negotiation.cc:769] Sending connection context
0924 03:09:22.082147 (+ 243us) client_negotiation.cc:240] Negotiation successful
0924 03:09:22.082550 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":306,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 03:09:22.085639 4322 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34503
I20260924 03:09:22.086004 4322 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:22.086755 4322 heartbeater.cc:511] Master 127.0.97.62:34503 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 03:09:22.088461 3999 ts_manager.cc:194] Registered new tserver with Master: ef71825cc92d4069b3f203f81d62cf05 (127.0.97.2:43259)
I20260924 03:09:22.089336 3999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 03:09:22.116868 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34503
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:22.458950 4327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:22.459232 4327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:22.459336 4327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:22.466496 4327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:22.466609 4327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:22.466722 4327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:09:22.477010 4327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:22.478379 4327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:22.479592 4327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:22.492509 4332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:22.492555 4335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:22.492647 4333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:22.493680 4327 server_base.cc:1034] running on GCE node
I20260924 03:09:22.494365 4327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:22.495381 4327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:22.496623 4327 hybrid_clock.cc:648] HybridClock initialized: now 1790219362496571 us; error 50 us; skew 500 ppm
I20260924 03:09:22.496963 4327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:22 dist-test-slave-5l98 krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:22.500451 4327 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:09:22.502254 4327 webserver.cc:460] Webserver started at http://127.0.97.3:35027/ using document root <none> and password file <none>
I20260924 03:09:22.502977 4327 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:22.503118 4327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:22.503444 4327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:22.506464 4327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "573e5645a7624ba8b2df751efe692c14"
format_stamp: "Formatted at 2026-09-24 03:09:22 on dist-test-slave-5l98"
server_key: "90f9be262b83e6405813f13d9be42e3c"
server_key_iv: "989d6e3d1d3de408d44e490b6ddc4913"
server_key_version: "encryptionkey@0"
I20260924 03:09:22.507277 4327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "573e5645a7624ba8b2df751efe692c14"
format_stamp: "Formatted at 2026-09-24 03:09:22 on dist-test-slave-5l98"
server_key: "90f9be262b83e6405813f13d9be42e3c"
server_key_iv: "989d6e3d1d3de408d44e490b6ddc4913"
server_key_version: "encryptionkey@0"
I20260924 03:09:22.512938 4327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:09:22.516707 4342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:22.518122 4327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:09:22.518307 4327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "573e5645a7624ba8b2df751efe692c14"
format_stamp: "Formatted at 2026-09-24 03:09:22 on dist-test-slave-5l98"
server_key: "90f9be262b83e6405813f13d9be42e3c"
server_key_iv: "989d6e3d1d3de408d44e490b6ddc4913"
server_key_version: "encryptionkey@0"
I20260924 03:09:22.518523 4327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:22.540699 4327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:22.545270 4327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:22.545616 4327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:22.546717 4327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:22.548069 4327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:22.548180 4327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:22.548339 4327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:22.548413 4327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:22.583843 4327 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38027
I20260924 03:09:22.583938 4454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38027 every 8 connection(s)
I20260924 03:09:22.585220 4327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:09:22.586980 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 4327
I20260924 03:09:22.587181 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:09:22.587579 388 external_mini_cluster.cc:1428] Setting key bad3940c01a9cc6a7239db17b1ce0416
I20260924 03:09:22.590345 4188 heartbeater.cc:503] Master 127.0.97.62:34503 was elected leader, sending a full tablet report...
Sep 24 03:09:22 dist-test-slave-5l98 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:22.607931 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:22.587866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37207 (local address 127.0.97.62:34503)
0924 03:09:22.588331 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:22.588338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:22.589759 (+ 1421us) server_negotiation.cc:407] Connection header received
0924 03:09:22.592305 (+ 2546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:22.592309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:22.592366 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:22.592458 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:22.594173 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:22.594949 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:22.596885 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:22.597329 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:22.597438 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:22.601591 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:22.601599 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:22.601606 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:22.601645 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:22.605003 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:22.605713 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:22.605717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:22.605719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:22.605838 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:22.606288 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:22.606291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:22.606293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:22.606560 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:22.606670 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:22.607481 (+ 811us) server_negotiation.cc:299] Negotiation successful
0924 03:09:22.607640 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":229,"threads_started":1}
I20260924 03:09:22.608937 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:22.588603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34503 (local address 127.0.97.3:37207)
0924 03:09:22.589569 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:22.589633 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:22.592048 (+ 2415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:22.592768 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:22.592778 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:22.593405 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:22.593999 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:22.594017 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:22.595159 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:22.595163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:22.596748 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:22.596761 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:22.597568 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:22.597573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:22.597823 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:22.598603 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:22.598627 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:22.601303 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:22.605146 (+ 3843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:22.605153 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:22.605167 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:22.605598 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:22.605930 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:22.605933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:22.605935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:22.606182 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:22.606671 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:22.606677 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:22.606992 (+ 315us) client_negotiation.cc:769] Sending connection context
0924 03:09:22.607280 (+ 288us) client_negotiation.cc:240] Negotiation successful
0924 03:09:22.607679 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":419,"threads_started":1}
I20260924 03:09:22.610298 4455 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34503
I20260924 03:09:22.610615 4455 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:22.611609 4455 heartbeater.cc:511] Master 127.0.97.62:34503 requested a full tablet report, sending...
I20260924 03:09:22.613377 3999 ts_manager.cc:194] Registered new tserver with Master: 573e5645a7624ba8b2df751efe692c14 (127.0.97.3:38027)
I20260924 03:09:22.614216 3999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37207
I20260924 03:09:22.616093 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:09:22 dist-test-slave-5l98 krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:09:22 dist-test-slave-5l98 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:09:22.667443 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:22.650684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60544 (local address 127.0.97.1:41863)
0924 03:09:22.651430 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:22.651441 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:22.651474 (+ 33us) server_negotiation.cc:407] Connection header received
0924 03:09:22.651703 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:22.651712 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:22.652006 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:22.652193 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:22.653741 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:22.654867 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:22.656225 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:22.656934 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:22.657080 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:22.661457 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:22.661468 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:22.661483 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:22.661531 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:22.664373 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:22.665207 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:22.665216 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:22.665219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:22.665361 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:22.665785 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:22.665789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:22.665791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:22.666312 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:22.666552 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:22.666923 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 03:09:22.667141 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":397,"threads_started":1}
Sep 24 03:09:22 dist-test-slave-5l98 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:22.687206 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:22.672264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47982 (local address 127.0.97.62:34503)
0924 03:09:22.672565 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:22.672571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:22.672824 (+ 253us) server_negotiation.cc:407] Connection header received
0924 03:09:22.673022 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:22.673025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:22.673177 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:22.673268 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:22.674510 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:22.675248 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:22.676857 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:22.677392 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:22.677503 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:22.681623 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:22.681628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:22.681631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:22.681665 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:22.684560 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:22.685208 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:22.685212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:22.685214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:22.685317 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:22.685692 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:22.685696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:22.685698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:22.686121 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:22.686304 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:22.686664 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 03:09:22.686901 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
W20260924 03:09:22.687734 3998 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47982
I20260924 03:09:22.688839 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 4060
I20260924 03:09:22.702863 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:22.689850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.0.97.1:41863)
0924 03:09:22.690044 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:22.690048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:22.690256 (+ 208us) server_negotiation.cc:407] Connection header received
0924 03:09:22.690378 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:22.690381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:22.690448 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:22.690526 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:22.691813 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:22.692618 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:22.693968 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:22.694378 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:22.694472 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:22.697113 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:22.697118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:22.697121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:22.697163 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:22.699832 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:22.700647 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:22.700651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:22.700654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:22.700841 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:22.701338 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:22.701341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:22.701343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:22.701600 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:22.701797 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:22.702248 (+ 451us) server_negotiation.cc:299] Negotiation successful
0924 03:09:22.702622 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 03:09:22.703503 4183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:22.788101 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 4060
I20260924 03:09:22.804352 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 4194
Sep 24 03:09:22 dist-test-slave-5l98 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 03:09:22.820878 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:22.805386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38052 (local address 127.0.97.2:43259)
0924 03:09:22.806000 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:22.806007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:22.806033 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:09:22.806092 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:22.806099 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:22.806328 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:22.806465 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:22.807605 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:22.808677 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:22.809980 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:22.810503 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:22.810598 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:22.815003 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:22.815015 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:22.815028 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:22.815062 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:22.817925 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:22.818773 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:22.818783 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:22.818786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:22.818911 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:22.819284 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:22.819291 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:22.819294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:22.819622 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:22.819931 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:22.820241 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 03:09:22.820544 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":332,"threads_started":1}
I20260924 03:09:22.822172 4317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:22.908654 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 4194
I20260924 03:09:22.924362 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 4327
Sep 24 03:09:22 dist-test-slave-5l98 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 03:09:22.941615 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:22.925422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42682 (local address 127.0.97.3:38027)
0924 03:09:22.926094 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:22.926101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:22.926129 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:09:22.926186 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:22.926193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:22.926442 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:22.926619 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:22.927925 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:22.929049 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:22.930318 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:22.930890 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:22.931045 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:22.935455 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:22.935467 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:22.935481 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:22.935520 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:22.938539 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:22.939215 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:22.939222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:22.939224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:22.939345 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:22.939839 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:22.939847 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:22.939853 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:22.940466 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:22.940839 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:22.941140 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 03:09:22.941334 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":378,"threads_started":1}
I20260924 03:09:22.942836 4450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:23.025887 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 4327
I20260924 03:09:23.041540 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 3967
I20260924 03:09:23.042860 4029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:23.152034 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 3967
2026-09-24T03:09:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3993 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:09:23 dist-test-slave-5l98 krb5kdc[4475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:09:23 dist-test-slave-5l98 krb5kdc[4475](info): set up 2 sockets
Sep 24 03:09:23 dist-test-slave-5l98 krb5kdc[4475](info): commencing operation
krb5kdc: starting...
W20260924 03:09:24.253252 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:09:24 dist-test-slave-5l98 krb5kdc[4475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:09:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:09:24.465595 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44559
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:24.792829 4488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:24.793121 4488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:24.793267 4488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:24.800752 4488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:24.800876 4488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:24.800922 4488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:24.800971 4488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:24.801010 4488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:09:24.811123 4488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:24.812503 4488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:24.813851 4488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:24.826558 4493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:24.826638 4496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:24.826614 4494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:24.827718 4488 server_base.cc:1034] running on GCE node
I20260924 03:09:24.828743 4488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:24.830292 4488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:24.831575 4488 hybrid_clock.cc:648] HybridClock initialized: now 1790219364831542 us; error 42 us; skew 500 ppm
I20260924 03:09:24.832033 4488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:24 dist-test-slave-5l98 krb5kdc[4475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:24.835336 4488 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:09:24.836544 4488 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:24.836651 4488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:24.836932 4488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:24.839869 4488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "b799f2af9f004d1298018b4296cc74bb"
format_stamp: "Formatted at 2026-09-24 03:09:24 on dist-test-slave-5l98"
server_key: "354858f49440c2d419df242e0d7921bc"
server_key_iv: "75805791adf306ffb6e009609d7e746f"
server_key_version: "encryptionkey@0"
I20260924 03:09:24.840752 4488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "b799f2af9f004d1298018b4296cc74bb"
format_stamp: "Formatted at 2026-09-24 03:09:24 on dist-test-slave-5l98"
server_key: "354858f49440c2d419df242e0d7921bc"
server_key_iv: "75805791adf306ffb6e009609d7e746f"
server_key_version: "encryptionkey@0"
I20260924 03:09:24.846618 4488 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260924 03:09:24.850436 4502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:24.852047 4488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:09:24.852192 4488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "b799f2af9f004d1298018b4296cc74bb"
format_stamp: "Formatted at 2026-09-24 03:09:24 on dist-test-slave-5l98"
server_key: "354858f49440c2d419df242e0d7921bc"
server_key_iv: "75805791adf306ffb6e009609d7e746f"
server_key_version: "encryptionkey@0"
I20260924 03:09:24.852391 4488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:24.866859 4488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:24.871268 4488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:24.871491 4488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:24.890478 4488 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44559
I20260924 03:09:24.890457 4553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44559 every 8 connection(s)
I20260924 03:09:24.891474 4488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:09:24.895432 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:24.895963 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 4488
I20260924 03:09:24.896188 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:09:24.897002 388 external_mini_cluster.cc:1428] Setting key 1f6272debe6ae8fe33f50e0427530b96
I20260924 03:09:24.908543 4554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb: Bootstrap starting.
Sep 24 03:09:24 dist-test-slave-5l98 krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:24.911635 4554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:24.912731 4554 log.cc:824] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:24.916234 4554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb: No bootstrap required, opened a new log
I20260924 03:09:24.920433 4557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:24.899181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.0.97.62:44559)
0924 03:09:24.899997 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:24.900008 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:24.900050 (+ 42us) server_negotiation.cc:407] Connection header received
0924 03:09:24.902075 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:24.902097 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:24.902519 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:24.902935 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:24.904422 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:24.906186 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:24.907519 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:24.908239 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:24.908433 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:24.912460 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:24.912469 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:24.912530 (+ 61us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:24.912589 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:24.916166 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:24.917020 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:24.917027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:24.917033 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:24.917188 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:24.917585 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:24.917591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:24.917594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:24.918176 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:24.918504 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:24.918941 (+ 437us) server_negotiation.cc:299] Negotiation successful
0924 03:09:24.919394 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":370,"threads_started":1}
I20260924 03:09:24.925316 4554 raft_consensus.cc:348] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b799f2af9f004d1298018b4296cc74bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44559 } }
I20260924 03:09:24.925729 4554 raft_consensus.cc:374] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:24.925838 4554 raft_consensus.cc:729] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b799f2af9f004d1298018b4296cc74bb, State: Initialized, Role: FOLLOWER
I20260924 03:09:24.926406 4554 consensus_queue.cc:260] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b799f2af9f004d1298018b4296cc74bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44559 } }
I20260924 03:09:24.926652 4554 raft_consensus.cc:388] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:09:24.926802 4554 raft_consensus.cc:482] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:09:24.926975 4554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:24.929718 4554 raft_consensus.cc:504] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b799f2af9f004d1298018b4296cc74bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44559 } }
I20260924 03:09:24.930290 4554 leader_election.cc:302] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b799f2af9f004d1298018b4296cc74bb; no voters:
I20260924 03:09:24.930891 4554 leader_election.cc:288] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:09:24.931066 4559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:24.931756 4559 raft_consensus.cc:686] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb [term 1 LEADER]: Becoming Leader. State: Replica: b799f2af9f004d1298018b4296cc74bb, State: Running, Role: LEADER
I20260924 03:09:24.932322 4559 consensus_queue.cc:237] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b799f2af9f004d1298018b4296cc74bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44559 } }
I20260924 03:09:24.933120 4554 sys_catalog.cc:564] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:09:24.935551 4560 sys_catalog.cc:455] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b799f2af9f004d1298018b4296cc74bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b799f2af9f004d1298018b4296cc74bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44559 } } }
I20260924 03:09:24.935348 4561 sys_catalog.cc:455] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b799f2af9f004d1298018b4296cc74bb. Latest consensus state: current_term: 1 leader_uuid: "b799f2af9f004d1298018b4296cc74bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b799f2af9f004d1298018b4296cc74bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44559 } } }
I20260924 03:09:24.935889 4560 sys_catalog.cc:458] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:24.935954 4561 sys_catalog.cc:458] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:24.936995 4565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:09:24.942891 4565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:09:24.952900 4565 catalog_manager.cc:1347] Generated new cluster ID: f35bd49009af4c6aada0b28943936b87
I20260924 03:09:24.953019 4565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:09:25.005048 4565 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:09:25.006948 4565 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:09:25.018486 4565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb: Generated new TSK 0
I20260924 03:09:25.019608 4565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 03:09:25.081727 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44559
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:25.427203 4580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:25.427512 4580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:25.427615 4580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:25.435329 4580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:25.435475 4580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:25.435635 4580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:09:25.446560 4580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:25.447881 4580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:25.449337 4580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:25.463152 4588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:25.463111 4586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:25.463198 4585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:25.466355 4580 server_base.cc:1034] running on GCE node
I20260924 03:09:25.467160 4580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:25.468214 4580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:25.469426 4580 hybrid_clock.cc:648] HybridClock initialized: now 1790219365469337 us; error 75 us; skew 500 ppm
I20260924 03:09:25.469808 4580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:25 dist-test-slave-5l98 krb5kdc[4475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:25.473244 4580 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:09:25.474568 4580 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:25.474661 4580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:25.475018 4580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:25.478073 4580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "094278a59c984970aca62af33e6b4c4f"
format_stamp: "Formatted at 2026-09-24 03:09:25 on dist-test-slave-5l98"
server_key: "53fd165e85c2556c4ae37fbb24c33d6b"
server_key_iv: "4b42261e4674be1fee128c967434e3e8"
server_key_version: "encryptionkey@0"
I20260924 03:09:25.478899 4580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "094278a59c984970aca62af33e6b4c4f"
format_stamp: "Formatted at 2026-09-24 03:09:25 on dist-test-slave-5l98"
server_key: "53fd165e85c2556c4ae37fbb24c33d6b"
server_key_iv: "4b42261e4674be1fee128c967434e3e8"
server_key_version: "encryptionkey@0"
I20260924 03:09:25.484232 4580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260924 03:09:25.488161 4594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:25.489878 4580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:09:25.490043 4580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "094278a59c984970aca62af33e6b4c4f"
format_stamp: "Formatted at 2026-09-24 03:09:25 on dist-test-slave-5l98"
server_key: "53fd165e85c2556c4ae37fbb24c33d6b"
server_key_iv: "4b42261e4674be1fee128c967434e3e8"
server_key_version: "encryptionkey@0"
I20260924 03:09:25.490348 4580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:25.528265 4580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:25.533025 4580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:25.533327 4580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:25.534436 4580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:25.535816 4580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:25.535907 4580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:25.536059 4580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:25.536136 4580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:25.571305 4580 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33121
I20260924 03:09:25.571383 4706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33121 every 8 connection(s)
I20260924 03:09:25.572201 4580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:09:25.576182 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 4580
I20260924 03:09:25.576406 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:09:25.577316 388 external_mini_cluster.cc:1428] Setting key 79d73c74afe87f4660c955910ee91741
Sep 24 03:09:25 dist-test-slave-5l98 krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:25.593408 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:25.574839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50373 (local address 127.0.97.62:44559)
0924 03:09:25.575286 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:25.575291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:25.576664 (+ 1373us) server_negotiation.cc:407] Connection header received
0924 03:09:25.578645 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:25.578651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:25.578741 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:25.578859 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:25.580379 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:25.581168 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:25.582785 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:25.583187 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:25.583322 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:25.587454 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:25.587459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:25.587462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:25.587495 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:25.590136 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:25.590965 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:25.590971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:25.590975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:25.591094 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:25.591509 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:25.591515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:25.591518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:25.591782 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:25.591932 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:25.592762 (+ 830us) server_negotiation.cc:299] Negotiation successful
0924 03:09:25.592991 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":209,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 03:09:25.594703 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:25.575520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44559 (local address 127.0.97.1:50373)
0924 03:09:25.576416 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:25.576498 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:25.578371 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:25.579103 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:25.579111 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:25.579632 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:25.580210 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:25.580227 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:25.581343 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:25.581347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:25.582625 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:25.582636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:25.583410 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:25.583418 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:25.583718 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:25.584535 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:25.584572 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:25.587214 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:25.590302 (+ 3088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:25.590312 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:25.590343 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:25.590848 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:25.591210 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:25.591215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:25.591219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:25.591396 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:25.591932 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:25.591942 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:25.592326 (+ 384us) client_negotiation.cc:769] Sending connection context
0924 03:09:25.592669 (+ 343us) client_negotiation.cc:240] Negotiation successful
0924 03:09:25.593211 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":351,"threads_started":1}
I20260924 03:09:25.596283 4707 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44559
I20260924 03:09:25.596704 4707 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:25.597645 4707 heartbeater.cc:511] Master 127.0.97.62:44559 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 03:09:25.600122 4519 ts_manager.cc:194] Registered new tserver with Master: 094278a59c984970aca62af33e6b4c4f (127.0.97.1:33121)
I20260924 03:09:25.601862 4519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 03:09:25.625244 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44559
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:25.961364 4713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:25.961669 4713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:25.961822 4713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:25.969300 4713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:25.969425 4713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:25.969607 4713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:09:25.979846 4713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:25.981312 4713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:25.982659 4713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:25.995994 4719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:25.996136 4718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:25.996062 4721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:25.996755 4713 server_base.cc:1034] running on GCE node
I20260924 03:09:25.997390 4713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:25.998360 4713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:25.999580 4713 hybrid_clock.cc:648] HybridClock initialized: now 1790219365999540 us; error 31 us; skew 500 ppm
I20260924 03:09:25.999950 4713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:26 dist-test-slave-5l98 krb5kdc[4475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:26.008926 4713 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:09:26.010149 4713 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:26.010247 4713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:26.010533 4713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:26.013633 4713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "f5416217a9e64e0494fb6bded4fffaa6"
format_stamp: "Formatted at 2026-09-24 03:09:26 on dist-test-slave-5l98"
server_key: "4906cb8150b03d18f9d1fde530fbbd8b"
server_key_iv: "6dfa8d028cdcad211260ce1b6731abd9"
server_key_version: "encryptionkey@0"
I20260924 03:09:26.014446 4713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f5416217a9e64e0494fb6bded4fffaa6"
format_stamp: "Formatted at 2026-09-24 03:09:26 on dist-test-slave-5l98"
server_key: "4906cb8150b03d18f9d1fde530fbbd8b"
server_key_iv: "6dfa8d028cdcad211260ce1b6731abd9"
server_key_version: "encryptionkey@0"
I20260924 03:09:26.020602 4713 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 03:09:26.024653 4727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:26.026347 4713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:09:26.026579 4713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "f5416217a9e64e0494fb6bded4fffaa6"
format_stamp: "Formatted at 2026-09-24 03:09:26 on dist-test-slave-5l98"
server_key: "4906cb8150b03d18f9d1fde530fbbd8b"
server_key_iv: "6dfa8d028cdcad211260ce1b6731abd9"
server_key_version: "encryptionkey@0"
I20260924 03:09:26.026964 4713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:26.067147 4713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:26.072048 4713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:26.072351 4713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:26.073463 4713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:26.074965 4713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:26.075057 4713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:26.075191 4713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:26.075276 4713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:26.114516 4713 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42199
I20260924 03:09:26.114596 4839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42199 every 8 connection(s)
I20260924 03:09:26.115408 4713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:09:26.116227 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 4713
I20260924 03:09:26.116418 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:09:26.116952 388 external_mini_cluster.cc:1428] Setting key 632ce1ab7a9a1732d3fbd7cf1ad197a1
Sep 24 03:09:26 dist-test-slave-5l98 krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 03:09:26.138269 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:26.118330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37019 (local address 127.0.97.62:44559)
0924 03:09:26.118792 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:26.118798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:26.119964 (+ 1166us) server_negotiation.cc:407] Connection header received
0924 03:09:26.121816 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:26.121821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:26.121911 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:26.122027 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:26.123524 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.124230 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.125902 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.126751 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.126898 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:26.130489 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:26.130499 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:26.130515 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:26.130561 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:26.134764 (+ 4203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:26.135757 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:26.135763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:26.135768 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:26.135933 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:26.136451 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:26.136457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:26.136461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:26.136895 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:26.137030 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:26.137805 (+ 775us) server_negotiation.cc:299] Negotiation successful
0924 03:09:26.137977 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":239,"threads_started":1}
I20260924 03:09:26.139505 4843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:26.118994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44559 (local address 127.0.97.2:37019)
0924 03:09:26.119749 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:26.119785 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:26.121554 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:26.122253 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:26.122261 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:26.122775 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:26.123344 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:26.123363 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.124393 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.124401 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:26.125740 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:26.125753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.127063 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.127068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:26.127256 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:26.127937 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:26.127959 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:26.130168 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:26.135039 (+ 4871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:26.135047 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:26.135061 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:26.135570 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:26.136095 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:26.136100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:26.136103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:26.136305 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:26.137059 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:26.137065 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:26.137408 (+ 343us) client_negotiation.cc:769] Sending connection context
0924 03:09:26.137719 (+ 311us) client_negotiation.cc:240] Negotiation successful
0924 03:09:26.138204 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":332,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 03:09:26.141534 4840 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44559
I20260924 03:09:26.141983 4840 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:26.142710 4840 heartbeater.cc:511] Master 127.0.97.62:44559 requested a full tablet report, sending...
I20260924 03:09:26.144409 4519 ts_manager.cc:194] Registered new tserver with Master: f5416217a9e64e0494fb6bded4fffaa6 (127.0.97.2:42199)
I20260924 03:09:26.145637 4519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 03:09:26.168673 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44559
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:26.525139 4846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:26.525461 4846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:26.525627 4846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:26.533073 4846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:26.533231 4846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:26.533413 4846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:09:26.543520 4846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:26.544988 4846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:26.546293 4846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:26.560021 4854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:26.560050 4852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:26.560189 4851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:26.566488 4846 server_base.cc:1034] running on GCE node
I20260924 03:09:26.567231 4846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:26.568259 4846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:26.569478 4846 hybrid_clock.cc:648] HybridClock initialized: now 1790219366569408 us; error 62 us; skew 500 ppm
I20260924 03:09:26.569813 4846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:26 dist-test-slave-5l98 krb5kdc[4475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:26.573412 4846 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:09:26.575052 4846 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:26.575168 4846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:26.575481 4846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:26.578912 4846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "29d256311ad5445db6303aa24b8c78ce"
format_stamp: "Formatted at 2026-09-24 03:09:26 on dist-test-slave-5l98"
server_key: "8cd45a0b5ef20405442fb370ad8c2f45"
server_key_iv: "085cc4595625dc0d0d476066409696fd"
server_key_version: "encryptionkey@0"
I20260924 03:09:26.579672 4846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "29d256311ad5445db6303aa24b8c78ce"
format_stamp: "Formatted at 2026-09-24 03:09:26 on dist-test-slave-5l98"
server_key: "8cd45a0b5ef20405442fb370ad8c2f45"
server_key_iv: "085cc4595625dc0d0d476066409696fd"
server_key_version: "encryptionkey@0"
I20260924 03:09:26.585549 4846 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 03:09:26.589385 4860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:26.590958 4846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:09:26.591112 4846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "29d256311ad5445db6303aa24b8c78ce"
format_stamp: "Formatted at 2026-09-24 03:09:26 on dist-test-slave-5l98"
server_key: "8cd45a0b5ef20405442fb370ad8c2f45"
server_key_iv: "085cc4595625dc0d0d476066409696fd"
server_key_version: "encryptionkey@0"
I20260924 03:09:26.591310 4846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:26.605472 4707 heartbeater.cc:503] Master 127.0.97.62:44559 was elected leader, sending a full tablet report...
I20260924 03:09:26.607657 4846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:26.612429 4846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:26.612785 4846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:26.613787 4846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:26.615146 4846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:26.615231 4846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:26.615386 4846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:26.615460 4846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:26.652899 4846 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37049
I20260924 03:09:26.652947 4972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37049 every 8 connection(s)
I20260924 03:09:26.653772 4846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:09:26.662345 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 4846
I20260924 03:09:26.662556 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:09:26.663074 388 external_mini_cluster.cc:1428] Setting key a6fe702174d82e2f6e05995a87a6056f
Sep 24 03:09:26 dist-test-slave-5l98 krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:26.676044 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:26.656543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48447 (local address 127.0.97.62:44559)
0924 03:09:26.657022 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:26.657026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:26.657870 (+ 844us) server_negotiation.cc:407] Connection header received
0924 03:09:26.659901 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:26.659907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:26.660004 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:26.660138 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:26.661757 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.662894 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.665001 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.665653 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.665767 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:26.669845 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:26.669850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:26.669854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:26.669889 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:26.672685 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:26.673558 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:26.673562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:26.673565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:26.673693 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:26.674292 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:26.674296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:26.674298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:26.674620 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:26.674740 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:26.675582 (+ 842us) server_negotiation.cc:299] Negotiation successful
0924 03:09:26.675759 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":262,"threads_started":1}
I20260924 03:09:26.677134 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:26.656968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44559 (local address 127.0.97.3:48447)
0924 03:09:26.657704 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:26.657742 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:26.659623 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:26.660378 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:26.660387 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:26.660984 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:26.661579 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:26.661598 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.663095 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.663099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:26.664832 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:26.664846 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.665901 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.665908 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:26.666210 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:26.667122 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:26.667150 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:26.669594 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:26.672823 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:26.672831 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:26.672845 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:26.673373 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:26.673834 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:26.673838 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:26.673840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:26.674142 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:26.674746 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:26.674753 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:26.675177 (+ 424us) client_negotiation.cc:769] Sending connection context
0924 03:09:26.675419 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 03:09:26.675852 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":292,"threads_started":1}
I20260924 03:09:26.678704 4973 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44559
I20260924 03:09:26.679164 4973 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:26.679893 4973 heartbeater.cc:511] Master 127.0.97.62:44559 requested a full tablet report, sending...
I20260924 03:09:26.681533 4518 ts_manager.cc:194] Registered new tserver with Master: 29d256311ad5445db6303aa24b8c78ce (127.0.97.3:37049)
I20260924 03:09:26.682454 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48447
I20260924 03:09:26.692303 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:09:26.710584 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:26.697403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39064 (local address 127.0.97.62:44559)
0924 03:09:26.697648 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:26.697654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:26.697994 (+ 340us) server_negotiation.cc:407] Connection header received
0924 03:09:26.698194 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:26.698198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:26.698266 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:26.698361 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:26.699962 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.700902 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.702208 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.702624 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.702718 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:26.705136 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:26.705140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:26.705144 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:26.705178 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:26.707819 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:26.708626 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:26.708630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:26.708632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:26.708747 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:26.709129 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:26.709132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:26.709134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:26.709472 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:26.709667 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:26.710011 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 03:09:26.710195 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:09:26.715631 4518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:09:26.722847 4518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:26.749758 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:26.737035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37049 (local address 127.0.0.1:34434)
0924 03:09:26.740890 (+ 3855us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:26.740909 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:26.741037 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:26.742149 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:26.742156 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:26.742192 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:26.742729 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:26.742739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.744703 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.744712 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:26.747054 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:26.747069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.748809 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.748816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:26.748999 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:26.749028 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 03:09:26.749118 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 03:09:26.749320 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3726,"mutex_wait_us":36,"thread_start_us":2248,"threads_started":1}
I20260924 03:09:26.749796 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:26.737591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52034 (local address 127.0.97.2:42199)
0924 03:09:26.738953 (+ 1362us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:26.738967 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:26.740644 (+ 1677us) server_negotiation.cc:407] Connection header received
0924 03:09:26.741075 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:26.741086 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:26.741325 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:26.741546 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:26.742672 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.744261 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.746692 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.748096 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.748234 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:26.748453 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:26.749269 (+ 816us) server_negotiation.cc:299] Negotiation successful
0924 03:09:26.749458 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":908,"thread_start_us":672,"threads_started":1}
I20260924 03:09:26.749967 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:26.738060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34434 (local address 127.0.97.3:37049)
0924 03:09:26.740554 (+ 2494us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:26.740564 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:26.741192 (+ 628us) server_negotiation.cc:407] Connection header received
0924 03:09:26.741293 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:26.741300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:26.741570 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:26.741793 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:26.742913 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.744541 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.747232 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.748283 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.748402 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:26.748644 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:26.749450 (+ 806us) server_negotiation.cc:299] Negotiation successful
0924 03:09:26.749636 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2279,"thread_start_us":2104,"threads_started":1}
I20260924 03:09:26.750586 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:26.736522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42199 (local address 127.0.0.1:52034)
0924 03:09:26.739687 (+ 3165us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:26.739738 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:26.740818 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:26.741879 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:26.741889 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:26.741925 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:26.742407 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:26.742426 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.744429 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.744433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:26.746523 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:26.746535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.748634 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.748644 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:26.748888 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:26.748943 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 03:09:26.749090 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 03:09:26.750138 (+ 1048us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2918,"mutex_wait_us":811,"thread_start_us":4069,"threads_started":1}
I20260924 03:09:26.751955 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:26.742901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33121 (local address 127.0.0.1:43608)
0924 03:09:26.743438 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:26.743455 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:26.743575 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:26.744620 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:26.744626 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:26.744655 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:26.745195 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:26.745206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.747148 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.747153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:26.749191 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:26.749207 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.751088 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.751096 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:26.751259 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:26.751304 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 03:09:26.751398 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 03:09:26.751548 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":3,"thread_start_us":277,"threads_started":1}
I20260924 03:09:26.752074 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:26.741370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43608 (local address 127.0.97.1:33121)
0924 03:09:26.742895 (+ 1525us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:26.742904 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:26.743572 (+ 668us) server_negotiation.cc:407] Connection header received
0924 03:09:26.743706 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:26.743714 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:26.743944 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:26.744103 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:26.745393 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.746992 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.749405 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.750881 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.751009 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:26.751216 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:26.751481 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 03:09:26.751645 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1358,"thread_start_us":1209,"threads_started":1}
I20260924 03:09:26.754565 4908 tablet_service.cc:1467] Processing CreateTablet for tablet 9f07be28e17e4ea39bf4bd79b5961777 (DEFAULT_TABLE table=test-table [id=cfbd38376ac94c3e87fd96f456dec277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:26.754565 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 9f07be28e17e4ea39bf4bd79b5961777 (DEFAULT_TABLE table=test-table [id=cfbd38376ac94c3e87fd96f456dec277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:26.755898 4908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f07be28e17e4ea39bf4bd79b5961777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:26.756978 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f07be28e17e4ea39bf4bd79b5961777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:26.758373 4642 tablet_service.cc:1467] Processing CreateTablet for tablet 9f07be28e17e4ea39bf4bd79b5961777 (DEFAULT_TABLE table=test-table [id=cfbd38376ac94c3e87fd96f456dec277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:26.759832 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f07be28e17e4ea39bf4bd79b5961777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:26.769006 4992 tablet_bootstrap.cc:492] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce: Bootstrap starting.
I20260924 03:09:26.772198 4993 tablet_bootstrap.cc:492] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6: Bootstrap starting.
I20260924 03:09:26.772629 4992 tablet_bootstrap.cc:654] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:26.774057 4992 log.cc:824] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:26.774857 4994 tablet_bootstrap.cc:492] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f: Bootstrap starting.
I20260924 03:09:26.776291 4993 tablet_bootstrap.cc:654] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:26.777757 4993 log.cc:824] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:26.778556 4994 tablet_bootstrap.cc:654] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:26.779529 4992 tablet_bootstrap.cc:492] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce: No bootstrap required, opened a new log
I20260924 03:09:26.779834 4992 ts_tablet_manager.cc:1397] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260924 03:09:26.779794 4994 log.cc:824] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:26.780485 4993 tablet_bootstrap.cc:492] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6: No bootstrap required, opened a new log
I20260924 03:09:26.780808 4993 ts_tablet_manager.cc:1397] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260924 03:09:26.782814 4994 tablet_bootstrap.cc:492] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f: No bootstrap required, opened a new log
I20260924 03:09:26.783135 4994 ts_tablet_manager.cc:1397] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.008s
I20260924 03:09:26.789044 4993 raft_consensus.cc:348] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5416217a9e64e0494fb6bded4fffaa6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42199 } } peers { permanent_uuid: "29d256311ad5445db6303aa24b8c78ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37049 } } peers { permanent_uuid: "094278a59c984970aca62af33e6b4c4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33121 } }
I20260924 03:09:26.789517 4993 raft_consensus.cc:374] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:26.789665 4993 raft_consensus.cc:729] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5416217a9e64e0494fb6bded4fffaa6, State: Initialized, Role: FOLLOWER
I20260924 03:09:26.790313 4993 consensus_queue.cc:260] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5416217a9e64e0494fb6bded4fffaa6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42199 } } peers { permanent_uuid: "29d256311ad5445db6303aa24b8c78ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37049 } } peers { permanent_uuid: "094278a59c984970aca62af33e6b4c4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33121 } }
I20260924 03:09:26.790917 4992 raft_consensus.cc:348] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5416217a9e64e0494fb6bded4fffaa6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42199 } } peers { permanent_uuid: "29d256311ad5445db6303aa24b8c78ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37049 } } peers { permanent_uuid: "094278a59c984970aca62af33e6b4c4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33121 } }
I20260924 03:09:26.791415 4992 raft_consensus.cc:374] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:26.791536 4992 raft_consensus.cc:729] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d256311ad5445db6303aa24b8c78ce, State: Initialized, Role: FOLLOWER
I20260924 03:09:26.792086 4840 heartbeater.cc:503] Master 127.0.97.62:44559 was elected leader, sending a full tablet report...
I20260924 03:09:26.792199 4992 consensus_queue.cc:260] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5416217a9e64e0494fb6bded4fffaa6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42199 } } peers { permanent_uuid: "29d256311ad5445db6303aa24b8c78ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37049 } } peers { permanent_uuid: "094278a59c984970aca62af33e6b4c4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33121 } }
I20260924 03:09:26.792930 4993 ts_tablet_manager.cc:1428] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260924 03:09:26.794008 4973 heartbeater.cc:503] Master 127.0.97.62:44559 was elected leader, sending a full tablet report...
I20260924 03:09:26.794926 4992 ts_tablet_manager.cc:1428] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce: Time spent starting tablet: real 0.015s user 0.017s sys 0.000s
I20260924 03:09:26.794764 4994 raft_consensus.cc:348] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5416217a9e64e0494fb6bded4fffaa6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42199 } } peers { permanent_uuid: "29d256311ad5445db6303aa24b8c78ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37049 } } peers { permanent_uuid: "094278a59c984970aca62af33e6b4c4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33121 } }
I20260924 03:09:26.795390 4994 raft_consensus.cc:374] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:26.795557 4994 raft_consensus.cc:729] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 094278a59c984970aca62af33e6b4c4f, State: Initialized, Role: FOLLOWER
I20260924 03:09:26.796236 4994 consensus_queue.cc:260] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5416217a9e64e0494fb6bded4fffaa6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42199 } } peers { permanent_uuid: "29d256311ad5445db6303aa24b8c78ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37049 } } peers { permanent_uuid: "094278a59c984970aca62af33e6b4c4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33121 } }
I20260924 03:09:26.798187 4994 ts_tablet_manager.cc:1428] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
W20260924 03:09:26.824793 4708 tablet.cc:2369] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:09:26.867513 4841 tablet.cc:2369] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:09:26.886797 5000 raft_consensus.cc:482] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:26.887238 5000 raft_consensus.cc:504] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5416217a9e64e0494fb6bded4fffaa6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42199 } } peers { permanent_uuid: "29d256311ad5445db6303aa24b8c78ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37049 } } peers { permanent_uuid: "094278a59c984970aca62af33e6b4c4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33121 } }
I20260924 03:09:26.889104 5000 leader_election.cc:288] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5416217a9e64e0494fb6bded4fffaa6 (127.0.97.2:42199), 29d256311ad5445db6303aa24b8c78ce (127.0.97.3:37049)
I20260924 03:09:26.898433 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:26.889893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48179 (local address 127.0.97.3:37049)
0924 03:09:26.890220 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:26.890230 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:26.890919 (+ 689us) server_negotiation.cc:407] Connection header received
0924 03:09:26.891254 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:26.891260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:26.891344 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:26.891413 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:26.892296 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.893505 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.896075 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.897419 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.897575 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:26.897707 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:26.898033 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 03:09:26.898168 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260924 03:09:26.898545 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:26.889699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37049 (local address 127.0.97.1:48179)
0924 03:09:26.890785 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:26.890811 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:26.891071 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:26.891566 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:26.891590 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:26.891618 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:26.892107 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:26.892121 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.893702 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.893706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:26.895883 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:26.895897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.897645 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.897652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:26.897841 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:26.897885 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 03:09:26.897996 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 03:09:26.898142 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"mutex_wait_us":104,"thread_start_us":740,"threads_started":1}
I20260924 03:09:26.898895 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:26.890346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35749 (local address 127.0.97.2:42199)
0924 03:09:26.891398 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:26.891405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:26.891425 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:09:26.891539 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:26.891544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:26.891614 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:26.891709 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:26.892353 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.893514 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.896375 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.897641 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.897785 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:26.897886 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:26.898262 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 03:09:26.898391 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":914}
I20260924 03:09:26.899629 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:26.889974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42199 (local address 127.0.97.1:35749)
0924 03:09:26.891154 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:26.891171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:26.891258 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:26.891778 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:26.891783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:26.891808 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:26.892185 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:26.892198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.893640 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.893646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:26.896217 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:26.896233 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.897878 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.897884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:26.898082 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:26.898114 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 03:09:26.899091 (+ 977us) client_negotiation.cc:240] Negotiation successful
0924 03:09:26.899256 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"mutex_wait_us":31,"thread_start_us":862,"threads_started":1}
I20260924 03:09:26.899914 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f07be28e17e4ea39bf4bd79b5961777" candidate_uuid: "094278a59c984970aca62af33e6b4c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d256311ad5445db6303aa24b8c78ce" is_pre_election: true
I20260924 03:09:26.900478 4928 raft_consensus.cc:2454] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 094278a59c984970aca62af33e6b4c4f in term 0.
I20260924 03:09:26.900712 4795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f07be28e17e4ea39bf4bd79b5961777" candidate_uuid: "094278a59c984970aca62af33e6b4c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5416217a9e64e0494fb6bded4fffaa6" is_pre_election: true
I20260924 03:09:26.901263 4795 raft_consensus.cc:2454] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 094278a59c984970aca62af33e6b4c4f in term 0.
I20260924 03:09:26.901345 4598 leader_election.cc:302] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 094278a59c984970aca62af33e6b4c4f, 29d256311ad5445db6303aa24b8c78ce; no voters:
I20260924 03:09:26.901821 5000 raft_consensus.cc:2781] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:26.901988 5000 raft_consensus.cc:482] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:26.902137 5000 raft_consensus.cc:3037] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:26.904356 5000 raft_consensus.cc:504] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5416217a9e64e0494fb6bded4fffaa6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42199 } } peers { permanent_uuid: "29d256311ad5445db6303aa24b8c78ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37049 } } peers { permanent_uuid: "094278a59c984970aca62af33e6b4c4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33121 } }
W20260924 03:09:26.905267 4974 tablet.cc:2369] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:09:26.905769 4795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f07be28e17e4ea39bf4bd79b5961777" candidate_uuid: "094278a59c984970aca62af33e6b4c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5416217a9e64e0494fb6bded4fffaa6"
I20260924 03:09:26.905776 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f07be28e17e4ea39bf4bd79b5961777" candidate_uuid: "094278a59c984970aca62af33e6b4c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d256311ad5445db6303aa24b8c78ce"
I20260924 03:09:26.906011 4928 raft_consensus.cc:3037] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:26.905987 4795 raft_consensus.cc:3037] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:26.906276 5000 leader_election.cc:288] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f [CANDIDATE]: Term 1 election: Requested vote from peers f5416217a9e64e0494fb6bded4fffaa6 (127.0.97.2:42199), 29d256311ad5445db6303aa24b8c78ce (127.0.97.3:37049)
I20260924 03:09:26.909292 4928 raft_consensus.cc:2454] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 094278a59c984970aca62af33e6b4c4f in term 1.
I20260924 03:09:26.909292 4795 raft_consensus.cc:2454] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 094278a59c984970aca62af33e6b4c4f in term 1.
I20260924 03:09:26.909776 4597 leader_election.cc:302] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 094278a59c984970aca62af33e6b4c4f, f5416217a9e64e0494fb6bded4fffaa6; no voters:
I20260924 03:09:26.910063 5000 raft_consensus.cc:2781] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:26.910808 5000 raft_consensus.cc:686] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f [term 1 LEADER]: Becoming Leader. State: Replica: 094278a59c984970aca62af33e6b4c4f, State: Running, Role: LEADER
I20260924 03:09:26.911331 5000 consensus_queue.cc:237] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5416217a9e64e0494fb6bded4fffaa6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42199 } } peers { permanent_uuid: "29d256311ad5445db6303aa24b8c78ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37049 } } peers { permanent_uuid: "094278a59c984970aca62af33e6b4c4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33121 } }
I20260924 03:09:26.916669 4519 catalog_manager.cc:5495] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 094278a59c984970aca62af33e6b4c4f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "094278a59c984970aca62af33e6b4c4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5416217a9e64e0494fb6bded4fffaa6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29d256311ad5445db6303aa24b8c78ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "094278a59c984970aca62af33e6b4c4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33121 } health_report { overall_health: HEALTHY } } }
I20260924 03:09:26.943787 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:26.937606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43622 (local address 127.0.97.1:33121)
0924 03:09:26.937772 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:26.937777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:26.937794 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:09:26.938162 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:26.938167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:26.938266 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:26.938401 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:26.938807 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.939759 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.941433 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:26.941843 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:26.941957 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:26.942580 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:26.943129 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:26.943246 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:26.943365 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 03:09:26.943446 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 03:09:26.952062 4928 raft_consensus.cc:1261] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce [term 1 FOLLOWER]: Refusing update from remote peer 094278a59c984970aca62af33e6b4c4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:26.952116 4795 raft_consensus.cc:1261] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6 [term 1 FOLLOWER]: Refusing update from remote peer 094278a59c984970aca62af33e6b4c4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:26.953115 5000 consensus_queue.cc:1035] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5416217a9e64e0494fb6bded4fffaa6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:26.953473 5003 consensus_queue.cc:1035] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d256311ad5445db6303aa24b8c78ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:26.979995 4519 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:26.980288 4519 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:26.983867 4519 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b799f2af9f004d1298018b4296cc74bb: Sending DeleteTablet for 3 replicas of tablet 9f07be28e17e4ea39bf4bd79b5961777
I20260924 03:09:26.985169 4908 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f07be28e17e4ea39bf4bd79b5961777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34434
I20260924 03:09:26.985230 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f07be28e17e4ea39bf4bd79b5961777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52034
I20260924 03:09:26.985630 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f07be28e17e4ea39bf4bd79b5961777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43608
I20260924 03:09:26.986938 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 4580
I20260924 03:09:27.016666 5018 tablet_replica.cc:330] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f: stopping tablet replica
I20260924 03:09:27.017434 5018 raft_consensus.cc:2229] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:09:27.018581 5018 raft_consensus.cc:2258] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:27.024348 5017 tablet_replica.cc:330] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6: stopping tablet replica
I20260924 03:09:27.024816 5017 raft_consensus.cc:2229] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:27.025375 5017 raft_consensus.cc:2258] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:27.027724 5017 ts_tablet_manager.cc:1905] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:27.028685 5018 ts_tablet_manager.cc:1905] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:27.033841 5016 tablet_replica.cc:330] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce: stopping tablet replica
I20260924 03:09:27.034417 5016 raft_consensus.cc:2229] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:27.035362 5017 ts_tablet_manager.cc:1918] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:27.035621 5017 log.cc:1196] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/9f07be28e17e4ea39bf4bd79b5961777
I20260924 03:09:27.036258 5017 ts_tablet_manager.cc:1939] T 9f07be28e17e4ea39bf4bd79b5961777 P f5416217a9e64e0494fb6bded4fffaa6: Deleting consensus metadata
Sep 24 03:09:27 dist-test-slave-5l98 krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:09:27.036990 5016 raft_consensus.cc:2258] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:27.037513 5018 ts_tablet_manager.cc:1918] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:27.037739 5018 log.cc:1196] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/9f07be28e17e4ea39bf4bd79b5961777
I20260924 03:09:27.038383 5018 ts_tablet_manager.cc:1939] T 9f07be28e17e4ea39bf4bd79b5961777 P 094278a59c984970aca62af33e6b4c4f: Deleting consensus metadata
I20260924 03:09:27.038292 4505 catalog_manager.cc:4841] TS f5416217a9e64e0494fb6bded4fffaa6 (127.0.97.2:42199): tablet 9f07be28e17e4ea39bf4bd79b5961777 (table test-table [id=cfbd38376ac94c3e87fd96f456dec277]) successfully deleted
I20260924 03:09:27.040884 4505 catalog_manager.cc:4841] TS 094278a59c984970aca62af33e6b4c4f (127.0.97.1:33121): tablet 9f07be28e17e4ea39bf4bd79b5961777 (table test-table [id=cfbd38376ac94c3e87fd96f456dec277]) successfully deleted
I20260924 03:09:27.043071 5016 ts_tablet_manager.cc:1905] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:27.045598 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:26.988621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43624 (local address 127.0.97.1:33121)
0924 03:09:26.988899 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:26.988909 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:27.012617 (+ 23708us) server_negotiation.cc:407] Connection header received
0924 03:09:27.013447 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:27.013457 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:27.013571 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:27.013685 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:27.018903 (+ 5218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:27.020589 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:27.026029 (+ 5440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:27.026688 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:27.026807 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:27.038615 (+ 11808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:27.038630 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:27.038651 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:27.038709 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:27.042098 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:27.042990 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:27.042999 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:27.043002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:27.043184 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:27.043653 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:27.043661 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:27.043668 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:27.044139 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:27.044419 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:27.044877 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 03:09:27.045144 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:09:27.046481 4702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:27.050951 5016 ts_tablet_manager.cc:1918] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:27.051117 5016 log.cc:1196] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/9f07be28e17e4ea39bf4bd79b5961777
I20260924 03:09:27.051561 5016 ts_tablet_manager.cc:1939] T 9f07be28e17e4ea39bf4bd79b5961777 P 29d256311ad5445db6303aa24b8c78ce: Deleting consensus metadata
I20260924 03:09:27.053184 4506 catalog_manager.cc:4841] TS 29d256311ad5445db6303aa24b8c78ce (127.0.97.3:37049): tablet 9f07be28e17e4ea39bf4bd79b5961777 (table test-table [id=cfbd38376ac94c3e87fd96f456dec277]) successfully deleted
I20260924 03:09:27.153283 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 4580
I20260924 03:09:27.170787 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 4713
Sep 24 03:09:27 dist-test-slave-5l98 krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 03:09:27.186383 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:27.171756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52042 (local address 127.0.97.2:42199)
0924 03:09:27.171979 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:27.171988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:27.172014 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:09:27.172073 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:27.172076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:27.172151 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:27.172280 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:27.173556 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:27.174280 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:27.175561 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:27.175977 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:27.176109 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:27.180221 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:27.180237 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:27.180254 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:27.180295 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:27.183384 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:27.184130 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:27.184139 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:27.184141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:27.184285 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:27.184778 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:27.184785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:27.184788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:27.185166 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:27.185460 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:27.185798 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 03:09:27.186034 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:09:27.187459 4835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:27.279647 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 4713
I20260924 03:09:27.297636 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 4846
Sep 24 03:09:27 dist-test-slave-5l98 krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 03:09:27.314741 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:27.298473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34450 (local address 127.0.97.3:37049)
0924 03:09:27.298660 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:27.298664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:27.298685 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:09:27.298899 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:27.298903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:27.298968 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:27.299062 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:27.300394 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:27.301271 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:27.302637 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:27.303120 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:27.303233 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:27.308134 (+ 4901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:27.308149 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:27.308166 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:27.308204 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:27.311277 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:27.312009 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:27.312016 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:27.312019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:27.312145 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:27.312586 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:27.312590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:27.312593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:27.312955 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:27.313308 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:27.313928 (+ 620us) server_negotiation.cc:299] Negotiation successful
0924 03:09:27.314453 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 03:09:27.315497 4968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:27.410900 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 4846
I20260924 03:09:27.428246 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 4488
I20260924 03:09:27.429477 4549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:27.541014 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 4488
2026-09-24T03:09:27Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4388 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T03:09:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:27Z Disabled control of system clock
I20260924 03:09:27.609170 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43199
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 03:09:27.939402 5027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:27.939755 5027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:27.939851 5027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:27.948252 5027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:27.948388 5027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:27.948433 5027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:27.948552 5027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:27.948637 5027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:09:27.959363 5027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43199
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:27.960880 5027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:27.962829 5027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:27.976809 5032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:27.976994 5035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:27.976969 5033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:27.977629 5027 server_base.cc:1034] running on GCE node
I20260924 03:09:27.978499 5027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:27.980093 5027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:27.981374 5027 hybrid_clock.cc:648] HybridClock initialized: now 1790219367981301 us; error 55 us; skew 500 ppm
I20260924 03:09:27.981837 5027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:09:27.984890 5027 webserver.cc:460] Webserver started at http://127.0.97.62:38553/ using document root <none> and password file <none>
I20260924 03:09:27.985692 5027 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:27.985816 5027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:27.986212 5027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:27.989493 5027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "da26ce04fe564e0bb4c60dd0ac341649"
format_stamp: "Formatted at 2026-09-24 03:09:27 on dist-test-slave-5l98"
server_key: "53055d2a4a1b67c90e41de75e02abc6b"
server_key_iv: "aa9b800b0a2c44625fe5c2b396d65bb8"
server_key_version: "encryptionkey@0"
I20260924 03:09:27.990388 5027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "da26ce04fe564e0bb4c60dd0ac341649"
format_stamp: "Formatted at 2026-09-24 03:09:27 on dist-test-slave-5l98"
server_key: "53055d2a4a1b67c90e41de75e02abc6b"
server_key_iv: "aa9b800b0a2c44625fe5c2b396d65bb8"
server_key_version: "encryptionkey@0"
I20260924 03:09:27.996332 5027 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 03:09:28.000212 5041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:28.001917 5027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:09:28.002115 5027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "da26ce04fe564e0bb4c60dd0ac341649"
format_stamp: "Formatted at 2026-09-24 03:09:27 on dist-test-slave-5l98"
server_key: "53055d2a4a1b67c90e41de75e02abc6b"
server_key_iv: "aa9b800b0a2c44625fe5c2b396d65bb8"
server_key_version: "encryptionkey@0"
I20260924 03:09:28.002347 5027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:28.009146 5027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:28.010062 5027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:28.010406 5027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:28.030262 5027 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43199
I20260924 03:09:28.030294 5092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43199 every 8 connection(s)
I20260924 03:09:28.031703 5027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:09:28.035883 5093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:28.039700 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 5027
I20260924 03:09:28.039893 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:09:28.040750 388 external_mini_cluster.cc:1428] Setting key 792f770060314de3246bf45fca009641
I20260924 03:09:28.048101 5093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649: Bootstrap starting.
I20260924 03:09:28.050841 5096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:28.043074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46932 (local address 127.0.97.62:43199)
0924 03:09:28.044189 (+ 1115us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:28.044204 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:28.044249 (+ 45us) server_negotiation.cc:407] Connection header received
0924 03:09:28.046780 (+ 2531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:28.046806 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:28.047275 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:28.047705 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:28.049136 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:28.049143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:28.049156 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:28.049173 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:28.049237 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:28.049253 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:28.049444 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:28.049553 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 03:09:28.049896 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":435,"threads_started":1}
I20260924 03:09:28.051862 5093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:28.053053 5093 log.cc:824] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:28.056149 5093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649: No bootstrap required, opened a new log
I20260924 03:09:28.063858 5093 raft_consensus.cc:348] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da26ce04fe564e0bb4c60dd0ac341649" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43199 } }
I20260924 03:09:28.064277 5093 raft_consensus.cc:374] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:28.064391 5093 raft_consensus.cc:729] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da26ce04fe564e0bb4c60dd0ac341649, State: Initialized, Role: FOLLOWER
I20260924 03:09:28.065092 5093 consensus_queue.cc:260] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da26ce04fe564e0bb4c60dd0ac341649" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43199 } }
I20260924 03:09:28.065326 5093 raft_consensus.cc:388] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:09:28.065483 5093 raft_consensus.cc:482] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:09:28.065687 5093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:28.067866 5093 raft_consensus.cc:504] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da26ce04fe564e0bb4c60dd0ac341649" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43199 } }
I20260924 03:09:28.068390 5093 leader_election.cc:302] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da26ce04fe564e0bb4c60dd0ac341649; no voters:
I20260924 03:09:28.069418 5093 leader_election.cc:288] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:09:28.069621 5098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:28.070999 5098 raft_consensus.cc:686] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649 [term 1 LEADER]: Becoming Leader. State: Replica: da26ce04fe564e0bb4c60dd0ac341649, State: Running, Role: LEADER
I20260924 03:09:28.071693 5093 sys_catalog.cc:564] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:09:28.071594 5098 consensus_queue.cc:237] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da26ce04fe564e0bb4c60dd0ac341649" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43199 } }
I20260924 03:09:28.074848 5100 sys_catalog.cc:455] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da26ce04fe564e0bb4c60dd0ac341649. Latest consensus state: current_term: 1 leader_uuid: "da26ce04fe564e0bb4c60dd0ac341649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da26ce04fe564e0bb4c60dd0ac341649" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43199 } } }
I20260924 03:09:28.075206 5100 sys_catalog.cc:458] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:28.075755 5099 sys_catalog.cc:455] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da26ce04fe564e0bb4c60dd0ac341649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da26ce04fe564e0bb4c60dd0ac341649" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43199 } } }
I20260924 03:09:28.076109 5099 sys_catalog.cc:458] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:28.076166 5106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:09:28.081787 5106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:09:28.091634 5106 catalog_manager.cc:1347] Generated new cluster ID: f4d3c55654374821a8e6f4a9417b930c
I20260924 03:09:28.091758 5106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:09:28.105050 5106 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:09:28.105492 5106 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:09:28.112851 5106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649: Generated new TSK 0
I20260924 03:09:28.113696 5106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:09:28.126395 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43199
--builtin_ntp_servers=127.0.97.20:41445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 03:09:28.511399 5117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:28.511729 5117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:28.511888 5117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:28.519526 5117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:28.519661 5117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:28.519834 5117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:09:28.529285 5117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:28.530638 5117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:28.532390 5117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:28.547492 5122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:28.547472 5125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:28.547449 5123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:28.548358 5117 server_base.cc:1034] running on GCE node
I20260924 03:09:28.549192 5117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:28.550256 5117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:28.551553 5117 hybrid_clock.cc:648] HybridClock initialized: now 1790219368551483 us; error 68 us; skew 500 ppm
I20260924 03:09:28.551898 5117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:09:28.554421 5117 webserver.cc:460] Webserver started at http://127.0.97.1:35593/ using document root <none> and password file <none>
I20260924 03:09:28.555160 5117 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:28.555297 5117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:28.555635 5117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:28.558713 5117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "df27513ea6964279b9040235ca654a75"
format_stamp: "Formatted at 2026-09-24 03:09:28 on dist-test-slave-5l98"
server_key: "064688dc9830e844c48592905f732cb5"
server_key_iv: "9fdd8f5921ad36ccc086ea050c2aa919"
server_key_version: "encryptionkey@0"
I20260924 03:09:28.559546 5117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "df27513ea6964279b9040235ca654a75"
format_stamp: "Formatted at 2026-09-24 03:09:28 on dist-test-slave-5l98"
server_key: "064688dc9830e844c48592905f732cb5"
server_key_iv: "9fdd8f5921ad36ccc086ea050c2aa919"
server_key_version: "encryptionkey@0"
I20260924 03:09:28.565322 5117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:09:28.569303 5131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:28.570814 5117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:09:28.571009 5117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "df27513ea6964279b9040235ca654a75"
format_stamp: "Formatted at 2026-09-24 03:09:28 on dist-test-slave-5l98"
server_key: "064688dc9830e844c48592905f732cb5"
server_key_iv: "9fdd8f5921ad36ccc086ea050c2aa919"
server_key_version: "encryptionkey@0"
I20260924 03:09:28.571260 5117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:28.577423 5117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:28.578285 5117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:28.578574 5117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:28.579610 5117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:28.581115 5117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:28.581245 5117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:28.581375 5117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:28.581483 5117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:28.616191 5117 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33405
I20260924 03:09:28.616283 5243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33405 every 8 connection(s)
I20260924 03:09:28.617530 5117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:09:28.620354 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 5117
I20260924 03:09:28.620630 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:09:28.621464 388 external_mini_cluster.cc:1428] Setting key 2c6ca2f6b21ac26eeeafb8ba7559069f
I20260924 03:09:28.624280 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43199
--builtin_ntp_servers=127.0.97.20:41445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 03:09:28.626610 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:28.620317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55181 (local address 127.0.97.62:43199)
0924 03:09:28.621002 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:28.621007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:28.622365 (+ 1358us) server_negotiation.cc:407] Connection header received
0924 03:09:28.624150 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:28.624156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:28.624227 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:28.624356 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:28.625765 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:28.625771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:28.625776 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:28.625795 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:28.625845 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:28.625863 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:28.626009 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:28.626236 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 03:09:28.626304 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":384,"threads_started":1}
I20260924 03:09:28.627553 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:28.621057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43199 (local address 127.0.97.1:55181)
0924 03:09:28.622119 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:28.622166 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:28.623882 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:28.624638 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:28.624646 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:28.624744 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:28.625458 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:09:28.625461 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:28.625498 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:09:28.625586 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:28.625994 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:28.626004 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:28.626012 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 03:09:28.626316 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 03:09:28.626589 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":327,"threads_started":1}
I20260924 03:09:28.629216 5244 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43199
I20260924 03:09:28.629577 5244 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:28.630263 5244 heartbeater.cc:511] Master 127.0.97.62:43199 requested a full tablet report, sending...
I20260924 03:09:28.632803 5058 ts_manager.cc:194] Registered new tserver with Master: df27513ea6964279b9040235ca654a75 (127.0.97.1:33405)
W20260924 03:09:28.957933 5248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:28.958226 5248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:28.958314 5248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:28.965965 5248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:28.966073 5248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:28.966185 5248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:09:28.975638 5248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:28.977026 5248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:28.978763 5248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:28.994115 5256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:28.994586 5253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:28.994735 5254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:28.998437 5248 server_base.cc:1034] running on GCE node
I20260924 03:09:28.999119 5248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:29.000227 5248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:29.001507 5248 hybrid_clock.cc:648] HybridClock initialized: now 1790219369001420 us; error 76 us; skew 500 ppm
I20260924 03:09:29.001912 5248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:09:29.004729 5248 webserver.cc:460] Webserver started at http://127.0.97.2:41543/ using document root <none> and password file <none>
I20260924 03:09:29.005589 5248 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:29.005745 5248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:29.006165 5248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:29.009863 5248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "dcc70fa0004c408c8733451760cd71d9"
format_stamp: "Formatted at 2026-09-24 03:09:29 on dist-test-slave-5l98"
server_key: "a513110fe01fb5d669480628dcbd44f6"
server_key_iv: "2a3ecbd9058466eef3938f7717839b65"
server_key_version: "encryptionkey@0"
I20260924 03:09:29.010797 5248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "dcc70fa0004c408c8733451760cd71d9"
format_stamp: "Formatted at 2026-09-24 03:09:29 on dist-test-slave-5l98"
server_key: "a513110fe01fb5d669480628dcbd44f6"
server_key_iv: "2a3ecbd9058466eef3938f7717839b65"
server_key_version: "encryptionkey@0"
I20260924 03:09:29.016908 5248 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260924 03:09:29.021144 5262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:29.022745 5248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:09:29.022939 5248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "dcc70fa0004c408c8733451760cd71d9"
format_stamp: "Formatted at 2026-09-24 03:09:29 on dist-test-slave-5l98"
server_key: "a513110fe01fb5d669480628dcbd44f6"
server_key_iv: "2a3ecbd9058466eef3938f7717839b65"
server_key_version: "encryptionkey@0"
I20260924 03:09:29.023216 5248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:29.029475 5248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:29.030346 5248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:29.030619 5248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:29.031639 5248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:29.033125 5248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:29.033250 5248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:29.033398 5248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:29.033490 5248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:29.070616 5248 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45557
I20260924 03:09:29.070669 5374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45557 every 8 connection(s)
I20260924 03:09:29.072045 5248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:09:29.076786 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 5248
I20260924 03:09:29.076972 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:09:29.077548 388 external_mini_cluster.cc:1428] Setting key 8f393b25ca359ffc43622c02f6976edc
I20260924 03:09:29.080924 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43199
--builtin_ntp_servers=127.0.97.20:41445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 03:09:29.082042 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:29.074994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35857 (local address 127.0.97.62:43199)
0924 03:09:29.075201 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:29.075205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:29.076848 (+ 1643us) server_negotiation.cc:407] Connection header received
0924 03:09:29.079085 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:29.079091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:29.079145 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:29.079249 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:29.080915 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:29.080923 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:29.080928 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:29.080955 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:29.081019 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:29.081060 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:29.081230 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:29.081620 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 03:09:29.081718 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 03:09:29.083186 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:29.075452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43199 (local address 127.0.97.2:35857)
0924 03:09:29.076658 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:29.076712 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:29.078809 (+ 2097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:29.079461 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:29.079469 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:29.079565 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:29.080474 (+ 909us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:09:29.080480 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:29.080558 (+ 78us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:09:29.080687 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:29.081294 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:29.081304 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:29.081311 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 03:09:29.081716 (+ 405us) client_negotiation.cc:240] Negotiation successful
0924 03:09:29.082007 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":441,"threads_started":1}
I20260924 03:09:29.084807 5375 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43199
I20260924 03:09:29.085140 5375 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:29.085843 5375 heartbeater.cc:511] Master 127.0.97.62:43199 requested a full tablet report, sending...
I20260924 03:09:29.087502 5058 ts_manager.cc:194] Registered new tserver with Master: dcc70fa0004c408c8733451760cd71d9 (127.0.97.2:45557)
W20260924 03:09:29.420919 5378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:29.421224 5378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:29.421325 5378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:29.429170 5378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:29.429286 5378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:29.429399 5378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:09:29.438768 5378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:29.440037 5378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:29.441944 5378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:29.455632 5383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:29.455931 5384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:29.456288 5386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:29.456826 5378 server_base.cc:1034] running on GCE node
I20260924 03:09:29.457401 5378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:29.458429 5378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:29.459688 5378 hybrid_clock.cc:648] HybridClock initialized: now 1790219369459639 us; error 45 us; skew 500 ppm
I20260924 03:09:29.460106 5378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:09:29.462810 5378 webserver.cc:460] Webserver started at http://127.0.97.3:37687/ using document root <none> and password file <none>
I20260924 03:09:29.463546 5378 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:29.463637 5378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:29.463996 5378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:29.466986 5378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "e01eb8e214764b8995bad414ae882674"
format_stamp: "Formatted at 2026-09-24 03:09:29 on dist-test-slave-5l98"
server_key: "22eb0fe835aa017718ff2c5e62913d97"
server_key_iv: "1e324124c2b0ce92f92b697cfe67160e"
server_key_version: "encryptionkey@0"
I20260924 03:09:29.467815 5378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e01eb8e214764b8995bad414ae882674"
format_stamp: "Formatted at 2026-09-24 03:09:29 on dist-test-slave-5l98"
server_key: "22eb0fe835aa017718ff2c5e62913d97"
server_key_iv: "1e324124c2b0ce92f92b697cfe67160e"
server_key_version: "encryptionkey@0"
I20260924 03:09:29.473536 5378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 03:09:29.477330 5392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:29.478807 5378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:09:29.478996 5378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "e01eb8e214764b8995bad414ae882674"
format_stamp: "Formatted at 2026-09-24 03:09:29 on dist-test-slave-5l98"
server_key: "22eb0fe835aa017718ff2c5e62913d97"
server_key_iv: "1e324124c2b0ce92f92b697cfe67160e"
server_key_version: "encryptionkey@0"
I20260924 03:09:29.479238 5378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:29.485286 5378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:29.486120 5378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:29.486423 5378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:29.487394 5378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:29.488947 5378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:29.489041 5378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:29.489194 5378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:29.489277 5378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:29.525177 5378 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36601
I20260924 03:09:29.525262 5504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36601 every 8 connection(s)
I20260924 03:09:29.526495 5378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:09:29.532791 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 5378
I20260924 03:09:29.532963 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:09:29.533631 388 external_mini_cluster.cc:1428] Setting key 08c125c21f802b5d32d5067448bb17bd
I20260924 03:09:29.535477 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:29.529277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46585 (local address 127.0.97.62:43199)
0924 03:09:29.529453 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:29.529457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:29.530806 (+ 1349us) server_negotiation.cc:407] Connection header received
0924 03:09:29.532605 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:29.532610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:29.532662 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:29.532753 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:29.534541 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:29.534547 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:29.534553 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:29.534576 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:29.534621 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:29.534635 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:29.534740 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:29.535118 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 03:09:29.535178 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 03:09:29.536650 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:29.529729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43199 (local address 127.0.97.3:46585)
0924 03:09:29.530620 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:29.530673 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:29.532216 (+ 1543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:29.533128 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:29.533142 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:29.533267 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:29.534138 (+ 871us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:09:29.534142 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:29.534200 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:09:29.534318 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:29.534750 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:29.534760 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:29.534769 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 03:09:29.535180 (+ 411us) client_negotiation.cc:240] Negotiation successful
0924 03:09:29.535508 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":271,"threads_started":1}
I20260924 03:09:29.538161 5505 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43199
I20260924 03:09:29.538420 5505 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:29.538967 5505 heartbeater.cc:511] Master 127.0.97.62:43199 requested a full tablet report, sending...
I20260924 03:09:29.540303 5058 ts_manager.cc:194] Registered new tserver with Master: e01eb8e214764b8995bad414ae882674 (127.0.97.3:36601)
I20260924 03:09:29.550285 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:09:29.557773 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:29.555275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.0.97.62:43199)
0924 03:09:29.555450 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:29.555455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:29.556016 (+ 561us) server_negotiation.cc:407] Connection header received
0924 03:09:29.556127 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:29.556131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:29.556185 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:29.556277 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:29.557239 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:29.557243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:29.557246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:29.557260 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:29.557290 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:29.557301 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:29.557405 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:29.557521 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 03:09:29.557567 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:09:29.562299 5058 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:09:29.569155 5058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:29.588146 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:29.582589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33405 (local address 127.0.0.1:45424)
0924 03:09:29.583884 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:29.583925 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:29.584171 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:29.585767 (+ 1596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:29.585774 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:29.585878 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:29.586680 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:09:29.586685 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:29.586715 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:09:29.586806 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:29.587350 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:29.587364 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:29.587372 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 03:09:29.587578 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 03:09:29.587675 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"thread_start_us":1730,"threads_started":1}
I20260924 03:09:29.588289 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:29.583413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58686 (local address 127.0.97.3:36601)
0924 03:09:29.584182 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:29.584189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:29.584751 (+ 562us) server_negotiation.cc:407] Connection header received
0924 03:09:29.584836 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:29.584846 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:29.585048 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:29.585196 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:29.587282 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:29.587294 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:29.587309 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:29.587329 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:29.587387 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:29.587409 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:29.587585 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:29.587791 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 03:09:29.587891 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":1097,"threads_started":1}
I20260924 03:09:29.589097 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:29.582792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45424 (local address 127.0.97.1:33405)
0924 03:09:29.583409 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:29.583420 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:29.584049 (+ 629us) server_negotiation.cc:407] Connection header received
0924 03:09:29.584313 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:29.584324 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:29.584588 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:29.584745 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:29.587016 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:29.587026 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:29.587041 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:29.587065 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:29.587122 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:29.587146 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:29.587258 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:29.588613 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0924 03:09:29.588712 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":324,"threads_started":1}
I20260924 03:09:29.589385 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:29.584702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45557 (local address 127.0.0.1:35012)
0924 03:09:29.585184 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:29.585203 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:29.585296 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:29.586733 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:29.586738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:29.586764 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:29.587432 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:09:29.587438 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:29.587455 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:09:29.587497 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:29.588733 (+ 1236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:29.588741 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:29.588750 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 03:09:29.588906 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 03:09:29.588951 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":702,"threads_started":1}
I20260924 03:09:29.589395 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:29.585224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35012 (local address 127.0.97.2:45557)
0924 03:09:29.585851 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:29.585859 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:29.585883 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:09:29.585951 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:29.585970 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:29.586153 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:29.586934 (+ 781us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:29.588404 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:29.588410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:29.588422 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:29.588453 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:29.588610 (+ 157us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:29.588635 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:29.588769 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:29.588896 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 03:09:29.588991 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":1172,"threads_started":1}
I20260924 03:09:29.588338 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:29.583072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36601 (local address 127.0.0.1:58686)
0924 03:09:29.584471 (+ 1399us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:29.584535 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:29.584638 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:29.586287 (+ 1649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:29.586291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:29.586313 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:29.587000 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:09:29.587004 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:29.587026 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:09:29.587078 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:29.587632 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:29.587637 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:29.587641 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:09:29.587827 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 03:09:29.587875 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1256,"thread_start_us":359,"threads_started":1}
I20260924 03:09:29.593405 5310 tablet_service.cc:1467] Processing CreateTablet for tablet 789d9cbdb14541c6862939eebbd9a58c (DEFAULT_TABLE table=test-table [id=7efb9d1264044224a56244fbdc8bfa1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:29.593361 5179 tablet_service.cc:1467] Processing CreateTablet for tablet 789d9cbdb14541c6862939eebbd9a58c (DEFAULT_TABLE table=test-table [id=7efb9d1264044224a56244fbdc8bfa1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:29.593587 5440 tablet_service.cc:1467] Processing CreateTablet for tablet 789d9cbdb14541c6862939eebbd9a58c (DEFAULT_TABLE table=test-table [id=7efb9d1264044224a56244fbdc8bfa1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:29.594834 5179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789d9cbdb14541c6862939eebbd9a58c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:29.594834 5310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789d9cbdb14541c6862939eebbd9a58c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:29.594914 5440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789d9cbdb14541c6862939eebbd9a58c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:29.606647 5523 tablet_bootstrap.cc:492] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674: Bootstrap starting.
I20260924 03:09:29.609316 5525 tablet_bootstrap.cc:492] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75: Bootstrap starting.
I20260924 03:09:29.612210 5523 tablet_bootstrap.cc:654] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:29.612949 5525 tablet_bootstrap.cc:654] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:29.613605 5523 log.cc:824] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:29.614099 5525 log.cc:824] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:29.617300 5523 tablet_bootstrap.cc:492] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674: No bootstrap required, opened a new log
I20260924 03:09:29.617631 5523 ts_tablet_manager.cc:1397] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260924 03:09:29.619292 5525 tablet_bootstrap.cc:492] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75: No bootstrap required, opened a new log
I20260924 03:09:29.619562 5525 ts_tablet_manager.cc:1397] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260924 03:09:29.615301 5524 tablet_bootstrap.cc:492] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9: Bootstrap starting.
I20260924 03:09:29.625387 5524 tablet_bootstrap.cc:654] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:29.625945 5523 raft_consensus.cc:348] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df27513ea6964279b9040235ca654a75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33405 } } peers { permanent_uuid: "dcc70fa0004c408c8733451760cd71d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45557 } } peers { permanent_uuid: "e01eb8e214764b8995bad414ae882674" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 } }
I20260924 03:09:29.626370 5523 raft_consensus.cc:374] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:29.626461 5523 raft_consensus.cc:729] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01eb8e214764b8995bad414ae882674, State: Initialized, Role: FOLLOWER
I20260924 03:09:29.626720 5524 log.cc:824] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:29.627295 5523 consensus_queue.cc:260] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df27513ea6964279b9040235ca654a75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33405 } } peers { permanent_uuid: "dcc70fa0004c408c8733451760cd71d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45557 } } peers { permanent_uuid: "e01eb8e214764b8995bad414ae882674" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 } }
I20260924 03:09:29.632450 5525 raft_consensus.cc:348] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df27513ea6964279b9040235ca654a75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33405 } } peers { permanent_uuid: "dcc70fa0004c408c8733451760cd71d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45557 } } peers { permanent_uuid: "e01eb8e214764b8995bad414ae882674" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 } }
I20260924 03:09:29.633118 5525 raft_consensus.cc:374] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:29.633242 5525 raft_consensus.cc:729] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df27513ea6964279b9040235ca654a75, State: Initialized, Role: FOLLOWER
I20260924 03:09:29.634209 5525 consensus_queue.cc:260] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df27513ea6964279b9040235ca654a75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33405 } } peers { permanent_uuid: "dcc70fa0004c408c8733451760cd71d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45557 } } peers { permanent_uuid: "e01eb8e214764b8995bad414ae882674" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 } }
I20260924 03:09:29.635390 5523 ts_tablet_manager.cc:1428] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674: Time spent starting tablet: real 0.018s user 0.010s sys 0.007s
I20260924 03:09:29.635727 5505 heartbeater.cc:503] Master 127.0.97.62:43199 was elected leader, sending a full tablet report...
I20260924 03:09:29.636274 5244 heartbeater.cc:503] Master 127.0.97.62:43199 was elected leader, sending a full tablet report...
I20260924 03:09:29.637576 5524 tablet_bootstrap.cc:492] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9: No bootstrap required, opened a new log
I20260924 03:09:29.637884 5524 ts_tablet_manager.cc:1397] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.008s
I20260924 03:09:29.639899 5525 ts_tablet_manager.cc:1428] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75: Time spent starting tablet: real 0.020s user 0.008s sys 0.011s
I20260924 03:09:29.646905 5524 raft_consensus.cc:348] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df27513ea6964279b9040235ca654a75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33405 } } peers { permanent_uuid: "dcc70fa0004c408c8733451760cd71d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45557 } } peers { permanent_uuid: "e01eb8e214764b8995bad414ae882674" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 } }
I20260924 03:09:29.647363 5524 raft_consensus.cc:374] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:29.647478 5524 raft_consensus.cc:729] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc70fa0004c408c8733451760cd71d9, State: Initialized, Role: FOLLOWER
I20260924 03:09:29.648070 5524 consensus_queue.cc:260] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df27513ea6964279b9040235ca654a75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33405 } } peers { permanent_uuid: "dcc70fa0004c408c8733451760cd71d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45557 } } peers { permanent_uuid: "e01eb8e214764b8995bad414ae882674" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 } }
I20260924 03:09:29.649406 5375 heartbeater.cc:503] Master 127.0.97.62:43199 was elected leader, sending a full tablet report...
I20260924 03:09:29.650060 5524 ts_tablet_manager.cc:1428] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
W20260924 03:09:29.778326 5506 tablet.cc:2369] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:09:29.824558 5376 tablet.cc:2369] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:09:29.870172 5245 tablet.cc:2369] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:09:30.083829 5530 raft_consensus.cc:482] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:30.084224 5530 raft_consensus.cc:504] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df27513ea6964279b9040235ca654a75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33405 } } peers { permanent_uuid: "dcc70fa0004c408c8733451760cd71d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45557 } } peers { permanent_uuid: "e01eb8e214764b8995bad414ae882674" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 } }
I20260924 03:09:30.086079 5530 leader_election.cc:288] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcc70fa0004c408c8733451760cd71d9 (127.0.97.2:45557), e01eb8e214764b8995bad414ae882674 (127.0.97.3:36601)
I20260924 03:09:30.090370 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:30.086537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38063 (local address 127.0.97.2:45557)
0924 03:09:30.086803 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:30.086814 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:30.087957 (+ 1143us) server_negotiation.cc:407] Connection header received
0924 03:09:30.088102 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:30.088111 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:30.088175 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:30.088274 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:30.089606 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:30.089611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:30.089616 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:30.089636 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:30.089672 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:30.089690 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:30.089801 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:30.089993 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 03:09:30.090076 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 03:09:30.090397 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:30.086427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45557 (local address 127.0.97.1:38063)
0924 03:09:30.087824 (+ 1397us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:30.087839 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:30.087959 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:30.088313 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:30.088318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:30.088339 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:30.089269 (+ 930us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:09:30.089271 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:30.089284 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:09:30.089320 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:30.089840 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:30.089846 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:30.089851 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:09:30.089979 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 03:09:30.090034 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"thread_start_us":1018,"threads_started":1}
I20260924 03:09:30.090585 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:30.086883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40289 (local address 127.0.97.3:36601)
0924 03:09:30.087301 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:30.087305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:30.088149 (+ 844us) server_negotiation.cc:407] Connection header received
0924 03:09:30.088308 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:30.088313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:30.088371 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:30.088513 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:30.089830 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:30.089834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:30.089839 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:30.089866 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:30.089906 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:30.089925 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:30.090041 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:30.090279 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 03:09:30.090340 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 03:09:30.091254 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:30.086501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36601 (local address 127.0.97.1:40289)
0924 03:09:30.087997 (+ 1496us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:30.088042 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:30.088163 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:30.088575 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:30.088584 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:30.088610 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:30.089544 (+ 934us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:09:30.089547 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:30.089565 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:09:30.089617 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:30.090125 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:30.090130 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:30.090134 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:09:30.090858 (+ 724us) client_negotiation.cc:240] Negotiation successful
0924 03:09:30.090908 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1319,"thread_start_us":498,"threads_started":1}
I20260924 03:09:30.091565 5330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789d9cbdb14541c6862939eebbd9a58c" candidate_uuid: "df27513ea6964279b9040235ca654a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc70fa0004c408c8733451760cd71d9" is_pre_election: true
I20260924 03:09:30.092097 5330 raft_consensus.cc:2454] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df27513ea6964279b9040235ca654a75 in term 0.
I20260924 03:09:30.092077 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789d9cbdb14541c6862939eebbd9a58c" candidate_uuid: "df27513ea6964279b9040235ca654a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01eb8e214764b8995bad414ae882674" is_pre_election: true
I20260924 03:09:30.092520 5460 raft_consensus.cc:2454] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df27513ea6964279b9040235ca654a75 in term 0.
I20260924 03:09:30.092883 5134 leader_election.cc:302] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcc70fa0004c408c8733451760cd71d9, df27513ea6964279b9040235ca654a75; no voters:
I20260924 03:09:30.093281 5530 raft_consensus.cc:2781] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:30.093408 5530 raft_consensus.cc:482] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:30.093556 5530 raft_consensus.cc:3037] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:30.095685 5530 raft_consensus.cc:504] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df27513ea6964279b9040235ca654a75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33405 } } peers { permanent_uuid: "dcc70fa0004c408c8733451760cd71d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45557 } } peers { permanent_uuid: "e01eb8e214764b8995bad414ae882674" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 } }
I20260924 03:09:30.096313 5530 leader_election.cc:288] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 [CANDIDATE]: Term 1 election: Requested vote from peers dcc70fa0004c408c8733451760cd71d9 (127.0.97.2:45557), e01eb8e214764b8995bad414ae882674 (127.0.97.3:36601)
I20260924 03:09:30.096894 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789d9cbdb14541c6862939eebbd9a58c" candidate_uuid: "df27513ea6964279b9040235ca654a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01eb8e214764b8995bad414ae882674"
I20260924 03:09:30.096905 5330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789d9cbdb14541c6862939eebbd9a58c" candidate_uuid: "df27513ea6964279b9040235ca654a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc70fa0004c408c8733451760cd71d9"
I20260924 03:09:30.097088 5330 raft_consensus.cc:3037] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:30.097134 5460 raft_consensus.cc:3037] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:30.099303 5460 raft_consensus.cc:2454] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df27513ea6964279b9040235ca654a75 in term 1.
I20260924 03:09:30.099296 5330 raft_consensus.cc:2454] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df27513ea6964279b9040235ca654a75 in term 1.
I20260924 03:09:30.099782 5134 leader_election.cc:302] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcc70fa0004c408c8733451760cd71d9, df27513ea6964279b9040235ca654a75; no voters:
I20260924 03:09:30.100200 5530 raft_consensus.cc:2781] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:30.101248 5530 raft_consensus.cc:686] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 [term 1 LEADER]: Becoming Leader. State: Replica: df27513ea6964279b9040235ca654a75, State: Running, Role: LEADER
I20260924 03:09:30.101879 5530 consensus_queue.cc:237] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df27513ea6964279b9040235ca654a75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33405 } } peers { permanent_uuid: "dcc70fa0004c408c8733451760cd71d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45557 } } peers { permanent_uuid: "e01eb8e214764b8995bad414ae882674" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 } }
I20260924 03:09:30.107030 5056 catalog_manager.cc:5495] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 reported cstate change: term changed from 0 to 1, leader changed from <none> to df27513ea6964279b9040235ca654a75 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "df27513ea6964279b9040235ca654a75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df27513ea6964279b9040235ca654a75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcc70fa0004c408c8733451760cd71d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01eb8e214764b8995bad414ae882674" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 } health_report { overall_health: UNKNOWN } } }
I20260924 03:09:30.170083 5541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:30.167154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45434 (local address 127.0.97.1:33405)
0924 03:09:30.167930 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:30.167934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:30.167951 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:09:30.168043 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:30.168046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:30.168121 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:30.168224 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:30.169241 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:30.169246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:30.169249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:30.169264 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:30.169307 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:30.169319 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:30.169488 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:30.169623 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 03:09:30.169670 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":544,"threads_started":1}
I20260924 03:09:30.179020 5330 raft_consensus.cc:1261] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9 [term 1 FOLLOWER]: Refusing update from remote peer df27513ea6964279b9040235ca654a75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:30.179150 5460 raft_consensus.cc:1261] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674 [term 1 FOLLOWER]: Refusing update from remote peer df27513ea6964279b9040235ca654a75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:30.180052 5530 consensus_queue.cc:1035] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01eb8e214764b8995bad414ae882674" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:30.180423 5534 consensus_queue.cc:1035] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc70fa0004c408c8733451760cd71d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:30.207037 5545 mvcc.cc:205] Tried to move back new op lower bound from 7332738540243701760 to 7332738539942060032. Current Snapshot: MvccSnapshot[applied={T|T < 7332738540243701760}]
I20260924 03:09:30.222010 5057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:30.222325 5057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:30.227021 5057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da26ce04fe564e0bb4c60dd0ac341649: Sending DeleteTablet for 3 replicas of tablet 789d9cbdb14541c6862939eebbd9a58c
I20260924 03:09:30.228400 5179 tablet_service.cc:1514] Processing DeleteTablet for tablet 789d9cbdb14541c6862939eebbd9a58c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:30 UTC) from {username='slave'} at 127.0.0.1:45424
I20260924 03:09:30.228363 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet 789d9cbdb14541c6862939eebbd9a58c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:30 UTC) from {username='slave'} at 127.0.0.1:58686
I20260924 03:09:30.229457 5310 tablet_service.cc:1514] Processing DeleteTablet for tablet 789d9cbdb14541c6862939eebbd9a58c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:30 UTC) from {username='slave'} at 127.0.0.1:35012
I20260924 03:09:30.230053 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 5117
I20260924 03:09:30.246351 5557 tablet_replica.cc:330] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9: stopping tablet replica
I20260924 03:09:30.248323 5541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:30.232833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45440 (local address 127.0.97.1:33405)
0924 03:09:30.233758 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:30.233767 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:30.244183 (+ 10416us) server_negotiation.cc:407] Connection header received
0924 03:09:30.244866 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:30.244877 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:30.244974 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:30.245069 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:30.247121 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:30.247132 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:30.247141 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:30.247173 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:30.247228 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:30.247252 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:30.247401 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:30.247713 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 03:09:30.247803 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665}
I20260924 03:09:30.249405 5557 raft_consensus.cc:2229] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:30.249769 5556 tablet_replica.cc:330] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75: stopping tablet replica
I20260924 03:09:30.250267 5556 raft_consensus.cc:2229] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:09:30.250576 5239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:30.251093 5557 raft_consensus.cc:2258] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:30.254117 5557 ts_tablet_manager.cc:1905] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:30.257826 5555 tablet_replica.cc:330] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674: stopping tablet replica
I20260924 03:09:30.258319 5555 raft_consensus.cc:2229] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:30.258775 5555 raft_consensus.cc:2258] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:30.260823 5555 ts_tablet_manager.cc:1905] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:30.260859 5557 ts_tablet_manager.cc:1918] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:30.261063 5557 log.cc:1196] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/789d9cbdb14541c6862939eebbd9a58c
I20260924 03:09:30.261538 5557 ts_tablet_manager.cc:1939] T 789d9cbdb14541c6862939eebbd9a58c P dcc70fa0004c408c8733451760cd71d9: Deleting consensus metadata
I20260924 03:09:30.263252 5044 catalog_manager.cc:4841] TS dcc70fa0004c408c8733451760cd71d9 (127.0.97.2:45557): tablet 789d9cbdb14541c6862939eebbd9a58c (table test-table [id=7efb9d1264044224a56244fbdc8bfa1b]) successfully deleted
I20260924 03:09:30.268452 5555 ts_tablet_manager.cc:1918] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:30.268671 5555 log.cc:1196] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/789d9cbdb14541c6862939eebbd9a58c
I20260924 03:09:30.269294 5555 ts_tablet_manager.cc:1939] T 789d9cbdb14541c6862939eebbd9a58c P e01eb8e214764b8995bad414ae882674: Deleting consensus metadata
I20260924 03:09:30.271489 5045 catalog_manager.cc:4841] TS e01eb8e214764b8995bad414ae882674 (127.0.97.3:36601): tablet 789d9cbdb14541c6862939eebbd9a58c (table test-table [id=7efb9d1264044224a56244fbdc8bfa1b]) successfully deleted
I20260924 03:09:30.354598 5556 raft_consensus.cc:2258] T 789d9cbdb14541c6862939eebbd9a58c P df27513ea6964279b9040235ca654a75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:30.355248 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 5117
W20260924 03:09:30.373093 5044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33405 (error 108)
I20260924 03:09:30.374092 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 5248
I20260924 03:09:30.376325 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:30.375171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33405 (local address 127.0.0.1:45456)
0924 03:09:30.375920 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:30.376077 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":495,"threads_started":1}
W20260924 03:09:30.376641 5044 catalog_manager.cc:4584] TS df27513ea6964279b9040235ca654a75 (127.0.97.1:33405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 789d9cbdb14541c6862939eebbd9a58c: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33405: connect: Connection refused (error 111)
I20260924 03:09:30.378983 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:30.375456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35024 (local address 127.0.97.2:45557)
0924 03:09:30.375675 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:30.375683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:30.376041 (+ 358us) server_negotiation.cc:407] Connection header received
0924 03:09:30.376264 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:30.376269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:30.376373 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:30.376644 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:30.377998 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:30.378003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:30.378007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:30.378028 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:30.378062 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:30.378074 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:30.378210 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:30.378617 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 03:09:30.378714 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 03:09:30.379771 5370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:30.430662 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:30.430134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33405 (local address 127.0.0.1:45472)
0924 03:09:30.430433 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:30.430516 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260924 03:09:30.432144 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:30.431660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33405 (local address 127.0.0.1:45486)
0924 03:09:30.431828 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:30.432001 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 03:09:30.470618 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 5248
I20260924 03:09:30.488981 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 5378
I20260924 03:09:30.492218 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:30.489850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58702 (local address 127.0.97.3:36601)
0924 03:09:30.490051 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:30.490055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:30.490075 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:09:30.490308 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:30.490313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:30.490406 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:30.490500 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:30.491551 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:30.491556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:30.491560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:30.491577 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:30.491618 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:30.491630 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:30.491747 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:30.491878 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 03:09:30.491934 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 03:09:30.493136 5500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:30.493392 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:30.492729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33405 (local address 127.0.0.1:45488)
0924 03:09:30.493074 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:30.493201 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 03:09:30.494786 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:30.494230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33405 (local address 127.0.0.1:45498)
0924 03:09:30.494471 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:30.494582 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 03:09:30.575778 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:30.575239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33405 (local address 127.0.0.1:45514)
0924 03:09:30.575475 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:30.575597 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 03:09:30.577088 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:30.576582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33405 (local address 127.0.0.1:45524)
0924 03:09:30.576837 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:30.576921 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260924 03:09:30.584846 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 5378
I20260924 03:09:30.602593 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 5027
I20260924 03:09:30.603859 5088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:30.710873 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 5027
2026-09-24T03:09:30Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3168 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 03:09:30.753253 388 mini_oidc.cc:152] Starting JWKS server
I20260924 03:09:30.755889 388 webserver.cc:460] Webserver started at https://127.0.0.1:42095/ using document root <none> and password file <none>
I20260924 03:09:30.756157 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 03:09:30.756896 388 webserver.cc:460] Webserver started at http://127.0.0.1:44093/ using document root <none> and password file <none>
2026-09-24T03:09:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:30Z Disabled control of system clock
I20260924 03:09:30.785713 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44891
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44891
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42095/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219327163871-388-0/testchainca.cert with env {}
W20260924 03:09:31.113953 5570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:31.114344 5570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:31.114512 5570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:31.121961 5570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:31.122120 5570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:31.122206 5570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:31.122318 5570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:31.122393 5570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 03:09:31.122488 5570 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 03:09:31.122565 5570 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42095/jwks.json
I20260924 03:09:31.132828 5570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44891
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42095/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219327163871-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:31.134346 5570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:31.135692 5570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:31.148306 5578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:31.148373 5575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:31.148365 5576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:31.148684 5570 server_base.cc:1034] running on GCE node
I20260924 03:09:31.150018 5570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:31.151669 5570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:31.153074 5570 hybrid_clock.cc:648] HybridClock initialized: now 1790219371152887 us; error 34 us; skew 500 ppm
I20260924 03:09:31.153520 5570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:09:31.160887 5570 webserver.cc:460] Webserver started at http://127.0.97.62:33919/ using document root <none> and password file <none>
I20260924 03:09:31.161785 5570 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:31.161893 5570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:31.162289 5570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:31.167383 5570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "e64c0dc30e1a4c40bad6339317783a1b"
format_stamp: "Formatted at 2026-09-24 03:09:31 on dist-test-slave-5l98"
server_key: "2b58f903878f83333b3eafa2683ec724"
server_key_iv: "0a66972f1c3fd8c7136fdcd655506489"
server_key_version: "encryptionkey@0"
I20260924 03:09:31.168349 5570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "e64c0dc30e1a4c40bad6339317783a1b"
format_stamp: "Formatted at 2026-09-24 03:09:31 on dist-test-slave-5l98"
server_key: "2b58f903878f83333b3eafa2683ec724"
server_key_iv: "0a66972f1c3fd8c7136fdcd655506489"
server_key_version: "encryptionkey@0"
I20260924 03:09:31.173828 5570 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 03:09:31.177881 5584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:31.179404 5570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:09:31.179642 5570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "e64c0dc30e1a4c40bad6339317783a1b"
format_stamp: "Formatted at 2026-09-24 03:09:31 on dist-test-slave-5l98"
server_key: "2b58f903878f83333b3eafa2683ec724"
server_key_iv: "0a66972f1c3fd8c7136fdcd655506489"
server_key_version: "encryptionkey@0"
I20260924 03:09:31.179919 5570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:31.219398 5570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:31.220429 5570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:31.220808 5570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:31.243119 5570 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44891
I20260924 03:09:31.243165 5635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44891 every 8 connection(s)
I20260924 03:09:31.244593 5570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:09:31.247179 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 5570
I20260924 03:09:31.247503 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:09:31.248136 388 external_mini_cluster.cc:1428] Setting key 0172d329ada5a919111485884214ed0e
I20260924 03:09:31.249261 5636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:31.264693 5636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b: Bootstrap starting.
I20260924 03:09:31.264729 5639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:31.250312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57796 (local address 127.0.97.62:44891)
0924 03:09:31.251315 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:31.251327 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:31.251378 (+ 51us) server_negotiation.cc:407] Connection header received
0924 03:09:31.254331 (+ 2953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:31.254356 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:31.254903 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:31.255277 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:31.256159 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:31.257652 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:31.259726 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:31.260435 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:31.260734 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:31.262126 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:31.262134 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:31.262151 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:31.262758 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:31.262849 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:31.262871 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:31.262961 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:31.263244 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 03:09:31.263629 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":359,"threads_started":1}
I20260924 03:09:31.269029 5636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:31.269395 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:09:31.270243 5636 log.cc:824] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:31.273633 5636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b: No bootstrap required, opened a new log
I20260924 03:09:31.282064 5636 raft_consensus.cc:348] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64c0dc30e1a4c40bad6339317783a1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44891 } }
I20260924 03:09:31.282526 5636 raft_consensus.cc:374] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:31.282657 5636 raft_consensus.cc:729] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e64c0dc30e1a4c40bad6339317783a1b, State: Initialized, Role: FOLLOWER
I20260924 03:09:31.283442 5636 consensus_queue.cc:260] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64c0dc30e1a4c40bad6339317783a1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44891 } }
I20260924 03:09:31.283748 5636 raft_consensus.cc:388] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:09:31.283913 5636 raft_consensus.cc:482] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:09:31.284145 5636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:31.286978 5636 raft_consensus.cc:504] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64c0dc30e1a4c40bad6339317783a1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44891 } }
I20260924 03:09:31.287557 5636 leader_election.cc:302] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e64c0dc30e1a4c40bad6339317783a1b; no voters:
I20260924 03:09:31.288197 5636 leader_election.cc:288] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:09:31.288319 5641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:31.289218 5641 raft_consensus.cc:686] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b [term 1 LEADER]: Becoming Leader. State: Replica: e64c0dc30e1a4c40bad6339317783a1b, State: Running, Role: LEADER
I20260924 03:09:31.289799 5641 consensus_queue.cc:237] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64c0dc30e1a4c40bad6339317783a1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44891 } }
I20260924 03:09:31.290700 5636 sys_catalog.cc:564] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:09:31.293581 5643 sys_catalog.cc:455] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e64c0dc30e1a4c40bad6339317783a1b. Latest consensus state: current_term: 1 leader_uuid: "e64c0dc30e1a4c40bad6339317783a1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64c0dc30e1a4c40bad6339317783a1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44891 } } }
I20260924 03:09:31.293979 5643 sys_catalog.cc:458] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:31.294498 5642 sys_catalog.cc:455] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e64c0dc30e1a4c40bad6339317783a1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64c0dc30e1a4c40bad6339317783a1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44891 } } }
I20260924 03:09:31.294852 5642 sys_catalog.cc:458] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:31.296345 5648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:09:31.302001 5648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:09:31.312294 5648 catalog_manager.cc:1347] Generated new cluster ID: 2684f469d7824f2e87beb4a4f2f4cc08
I20260924 03:09:31.312451 5648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:09:31.321573 5648 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:09:31.322925 5648 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:09:31.342265 5648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e64c0dc30e1a4c40bad6339317783a1b: Generated new TSK 0
I20260924 03:09:31.343425 5648 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 03:09:31.384186 5666 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 03:09:31.384383 5666 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 03:09:31.385648 5639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:31.361168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57802 (local address 127.0.97.62:44891)
0924 03:09:31.361380 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:31.361386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:31.361994 (+ 608us) server_negotiation.cc:407] Connection header received
0924 03:09:31.362284 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:31.362290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:31.362400 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:31.362575 (+ 175us) server_negotiation.cc:226] Negotiated authn=JWT
0924 03:09:31.363245 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:31.364537 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:31.366226 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:31.366948 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:31.367078 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:31.367730 (+ 652us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 03:09:31.384873 (+ 17143us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 03:09:31.385258 (+ 385us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57802: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 03:09:31.385782 5639 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57802: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 03:09:31.387238 5665 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:31.361104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44891 (local address 127.0.0.1:57802)
0924 03:09:31.361868 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:31.361896 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:31.362051 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:31.362642 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:31.362646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:31.362657 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0924 03:09:31.363031 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:31.363043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:31.364741 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:31.364746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:31.366056 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:31.366069 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:31.367126 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:31.367186 (+ 60us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:31.367319 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:31.367387 (+ 68us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 03:09:31.385188 (+ 17801us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 03:09:31.385515 (+ 327us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:44891: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":442,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:44891: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 03:09:31.389570 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 5570
I20260924 03:09:31.390837 5631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:31.487489 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 5570
2026-09-24T03:09:31Z chronyd exiting
I20260924 03:09:31.539374 388 test_util.cc:175] -----------------------------------------------
I20260924 03:09:31.539563 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219327163871-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (791 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 03:09:31.543341 388 mini_oidc.cc:152] Starting JWKS server
I20260924 03:09:31.545220 388 webserver.cc:460] Webserver started at https://127.0.0.1:35531/ using document root <none> and password file <none>
I20260924 03:09:31.545455 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 03:09:31.546108 388 webserver.cc:460] Webserver started at http://127.0.0.1:41807/ using document root <none> and password file <none>
2026-09-24T03:09:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:31Z Disabled control of system clock
I20260924 03:09:31.574329 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35247
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35247
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35531/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219327163871-388-0/testchainca.cert with env {}
W20260924 03:09:31.902769 5673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:31.903090 5673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:31.903239 5673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:31.910646 5673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:31.910773 5673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:31.910812 5673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:31.910946 5673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:31.911018 5673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 03:09:31.911103 5673 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 03:09:31.911166 5673 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35531/jwks.json
I20260924 03:09:31.921228 5673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35247
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35531/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219327163871-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:31.922640 5673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:31.924012 5673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:31.936409 5678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:31.937047 5679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:31.937227 5681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:31.937633 5673 server_base.cc:1034] running on GCE node
I20260924 03:09:31.938388 5673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:31.939913 5673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:31.941144 5673 hybrid_clock.cc:648] HybridClock initialized: now 1790219371941096 us; error 38 us; skew 500 ppm
I20260924 03:09:31.941545 5673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:09:31.944042 5673 webserver.cc:460] Webserver started at http://127.0.97.62:40893/ using document root <none> and password file <none>
I20260924 03:09:31.944834 5673 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:31.944926 5673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:31.945295 5673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:31.948228 5673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "7cbfebb3da264b38b7d89bffdb02366a"
format_stamp: "Formatted at 2026-09-24 03:09:31 on dist-test-slave-5l98"
server_key: "7ddddc093c0c5cfaa6c4669e77dfa8f1"
server_key_iv: "1027fee642f4ca06a8c738974f1ec038"
server_key_version: "encryptionkey@0"
I20260924 03:09:31.949178 5673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "7cbfebb3da264b38b7d89bffdb02366a"
format_stamp: "Formatted at 2026-09-24 03:09:31 on dist-test-slave-5l98"
server_key: "7ddddc093c0c5cfaa6c4669e77dfa8f1"
server_key_iv: "1027fee642f4ca06a8c738974f1ec038"
server_key_version: "encryptionkey@0"
I20260924 03:09:31.954551 5673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:09:31.958474 5687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:31.960013 5673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:09:31.960212 5673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "7cbfebb3da264b38b7d89bffdb02366a"
format_stamp: "Formatted at 2026-09-24 03:09:31 on dist-test-slave-5l98"
server_key: "7ddddc093c0c5cfaa6c4669e77dfa8f1"
server_key_iv: "1027fee642f4ca06a8c738974f1ec038"
server_key_version: "encryptionkey@0"
I20260924 03:09:31.960434 5673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:31.981313 5673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:31.982195 5673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:31.982491 5673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:32.002588 5738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35247 every 8 connection(s)
I20260924 03:09:32.002607 5673 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35247
I20260924 03:09:32.004168 5673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:09:32.005625 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 5673
I20260924 03:09:32.005844 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:09:32.006652 388 external_mini_cluster.cc:1428] Setting key 57f7f623162676d08cee4cb45df582db
I20260924 03:09:32.009970 5739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:32.020744 5742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:32.008893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58158 (local address 127.0.97.62:35247)
0924 03:09:32.010017 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:32.010062 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:32.010116 (+ 54us) server_negotiation.cc:407] Connection header received
0924 03:09:32.011686 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:32.011707 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:32.012148 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:32.012635 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:32.013182 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:32.014484 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:32.015765 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:32.016437 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:32.016681 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:32.017932 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:32.018098 (+ 166us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:32.018122 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:32.018753 (+ 631us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:32.018833 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:32.018856 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:32.019109 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:32.019346 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 03:09:32.019692 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":412,"threads_started":1}
I20260924 03:09:32.021066 5739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a: Bootstrap starting.
I20260924 03:09:32.024752 5739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:32.024940 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:09:32.025795 5739 log.cc:824] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:32.029313 5739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a: No bootstrap required, opened a new log
I20260924 03:09:32.037833 5739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbfebb3da264b38b7d89bffdb02366a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35247 } }
I20260924 03:09:32.038264 5739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:32.038384 5739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cbfebb3da264b38b7d89bffdb02366a, State: Initialized, Role: FOLLOWER
I20260924 03:09:32.038977 5739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbfebb3da264b38b7d89bffdb02366a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35247 } }
I20260924 03:09:32.039219 5739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:09:32.039368 5739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:09:32.039547 5739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:32.041821 5739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbfebb3da264b38b7d89bffdb02366a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35247 } }
I20260924 03:09:32.042351 5739 leader_election.cc:302] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cbfebb3da264b38b7d89bffdb02366a; no voters:
I20260924 03:09:32.042932 5739 leader_election.cc:288] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:09:32.043157 5744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:32.044202 5744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a [term 1 LEADER]: Becoming Leader. State: Replica: 7cbfebb3da264b38b7d89bffdb02366a, State: Running, Role: LEADER
I20260924 03:09:32.044914 5744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbfebb3da264b38b7d89bffdb02366a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35247 } }
I20260924 03:09:32.045295 5739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:09:32.048296 5745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cbfebb3da264b38b7d89bffdb02366a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbfebb3da264b38b7d89bffdb02366a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35247 } } }
I20260924 03:09:32.047966 5746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cbfebb3da264b38b7d89bffdb02366a. Latest consensus state: current_term: 1 leader_uuid: "7cbfebb3da264b38b7d89bffdb02366a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbfebb3da264b38b7d89bffdb02366a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35247 } } }
I20260924 03:09:32.048728 5745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:32.048761 5746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:32.050331 5753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:09:32.055558 5753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:09:32.065716 5753 catalog_manager.cc:1347] Generated new cluster ID: 9528d0e3c49d48d3b7b6173c7ddca30a
I20260924 03:09:32.065845 5753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:09:32.091809 5753 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:09:32.093636 5753 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:09:32.103976 5753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cbfebb3da264b38b7d89bffdb02366a: Generated new TSK 0
I20260924 03:09:32.105190 5753 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 03:09:32.154727 5769 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 03:09:32.154975 5769 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 03:09:32.156144 5742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:32.134346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58160 (local address 127.0.97.62:35247)
0924 03:09:32.134577 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:32.134585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:32.134964 (+ 379us) server_negotiation.cc:407] Connection header received
0924 03:09:32.135209 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:32.135215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:32.135335 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:32.135540 (+ 205us) server_negotiation.cc:226] Negotiated authn=JWT
0924 03:09:32.135950 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:32.137319 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:32.138755 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:32.139818 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:32.139978 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:32.140578 (+ 600us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 03:09:32.155412 (+ 14834us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 03:09:32.155828 (+ 416us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58160: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 03:09:32.156282 5742 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58160: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 03:09:32.156332 5768 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:32.134162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35247 (local address 127.0.0.1:58160)
0924 03:09:32.134852 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:32.134868 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:32.135006 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:32.135548 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:32.135552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:32.135561 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 03:09:32.135802 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:32.135809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:32.137489 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:32.137494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:32.138602 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:32.138614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:32.139961 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:32.140009 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:32.140153 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:32.140186 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 03:09:32.155716 (+ 15530us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 03:09:32.155951 (+ 235us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:35247: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":417,"threads_started":1}
I20260924 03:09:32.158083 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 5673
I20260924 03:09:32.159441 5734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:32.272198 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 5673
2026-09-24T03:09:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (786 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 03:09:32.330220 388 mini_oidc.cc:152] Starting JWKS server
I20260924 03:09:32.332049 388 webserver.cc:460] Webserver started at https://127.0.0.1:44001/ using document root <none> and password file <none>
I20260924 03:09:32.332244 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 03:09:32.332891 388 webserver.cc:460] Webserver started at http://127.0.0.1:45267/ using document root <none> and password file <none>
2026-09-24T03:09:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:32Z Disabled control of system clock
I20260924 03:09:32.360999 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38819
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38819
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44001/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 03:09:32.693397 5776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:32.693763 5776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:32.693919 5776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:32.701805 5776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:32.701969 5776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:32.702044 5776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:32.702123 5776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:32.702184 5776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 03:09:32.702273 5776 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 03:09:32.702351 5776 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44001/jwks.json
I20260924 03:09:32.711617 5776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38819
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44001/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:32.713066 5776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:32.714884 5776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:32.726791 5781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:32.726850 5782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:32.726917 5784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:32.727404 5776 server_base.cc:1034] running on GCE node
I20260924 03:09:32.728178 5776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:32.729763 5776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:32.731043 5776 hybrid_clock.cc:648] HybridClock initialized: now 1790219372731044 us; error 108 us; skew 500 ppm
I20260924 03:09:32.731443 5776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:09:32.734150 5776 webserver.cc:460] Webserver started at http://127.0.97.62:37871/ using document root <none> and password file <none>
I20260924 03:09:32.734899 5776 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:32.735047 5776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:32.735383 5776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:32.738487 5776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "4994c0392df844f7a85f2cf9bde9a904"
format_stamp: "Formatted at 2026-09-24 03:09:32 on dist-test-slave-5l98"
server_key: "33b1ff878825d3020c28f0a0c112312e"
server_key_iv: "a6d4e47805cf22ef85c42eeab9c57a5f"
server_key_version: "encryptionkey@0"
I20260924 03:09:32.739337 5776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "4994c0392df844f7a85f2cf9bde9a904"
format_stamp: "Formatted at 2026-09-24 03:09:32 on dist-test-slave-5l98"
server_key: "33b1ff878825d3020c28f0a0c112312e"
server_key_iv: "a6d4e47805cf22ef85c42eeab9c57a5f"
server_key_version: "encryptionkey@0"
I20260924 03:09:32.745159 5776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 03:09:32.748937 5790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:32.750519 5776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:09:32.750747 5776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "4994c0392df844f7a85f2cf9bde9a904"
format_stamp: "Formatted at 2026-09-24 03:09:32 on dist-test-slave-5l98"
server_key: "33b1ff878825d3020c28f0a0c112312e"
server_key_iv: "a6d4e47805cf22ef85c42eeab9c57a5f"
server_key_version: "encryptionkey@0"
I20260924 03:09:32.751030 5776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:32.769554 5776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:32.770557 5776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:32.770862 5776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:32.790269 5776 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38819
I20260924 03:09:32.790293 5841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38819 every 8 connection(s)
I20260924 03:09:32.791661 5776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:09:32.793170 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 5776
I20260924 03:09:32.793388 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:09:32.794011 388 external_mini_cluster.cc:1428] Setting key 199bd5ada20ff9282602da8aeb381b04
I20260924 03:09:32.797278 5842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:32.809473 5845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:32.796167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.0.97.62:38819)
0924 03:09:32.797325 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:32.797338 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:32.797384 (+ 46us) server_negotiation.cc:407] Connection header received
0924 03:09:32.799143 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:32.799166 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:32.799678 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:32.800060 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:32.800739 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:32.802186 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:32.804146 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:32.805026 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:32.805193 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:32.806545 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:32.806553 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:32.806573 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:32.807251 (+ 678us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:32.807366 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:32.807399 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:32.807495 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:32.807855 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 03:09:32.808274 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":442,"threads_started":1}
I20260924 03:09:32.810588 5842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904: Bootstrap starting.
I20260924 03:09:32.813824 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:09:32.814253 5842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:32.815297 5842 log.cc:824] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:32.818562 5842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904: No bootstrap required, opened a new log
I20260924 03:09:32.826263 5842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4994c0392df844f7a85f2cf9bde9a904" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38819 } }
I20260924 03:09:32.826766 5842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:32.826957 5842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4994c0392df844f7a85f2cf9bde9a904, State: Initialized, Role: FOLLOWER
I20260924 03:09:32.827654 5842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4994c0392df844f7a85f2cf9bde9a904" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38819 } }
I20260924 03:09:32.827915 5842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:09:32.828058 5842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:09:32.828248 5842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:32.830631 5842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4994c0392df844f7a85f2cf9bde9a904" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38819 } }
I20260924 03:09:32.831194 5842 leader_election.cc:302] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4994c0392df844f7a85f2cf9bde9a904; no voters:
I20260924 03:09:32.831780 5842 leader_election.cc:288] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:09:32.831991 5847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:32.832856 5847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904 [term 1 LEADER]: Becoming Leader. State: Replica: 4994c0392df844f7a85f2cf9bde9a904, State: Running, Role: LEADER
I20260924 03:09:32.833781 5847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4994c0392df844f7a85f2cf9bde9a904" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38819 } }
I20260924 03:09:32.834264 5842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:09:32.837095 5848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4994c0392df844f7a85f2cf9bde9a904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4994c0392df844f7a85f2cf9bde9a904" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38819 } } }
I20260924 03:09:32.837508 5848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:32.838073 5849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4994c0392df844f7a85f2cf9bde9a904. Latest consensus state: current_term: 1 leader_uuid: "4994c0392df844f7a85f2cf9bde9a904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4994c0392df844f7a85f2cf9bde9a904" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38819 } } }
I20260924 03:09:32.838439 5849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:32.841751 5856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:09:32.846935 5856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:09:32.856395 5856 catalog_manager.cc:1347] Generated new cluster ID: 48d36e5097fd4bd39b38c0c6880c7aca
I20260924 03:09:32.856606 5856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:09:32.876109 5856 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:09:32.877599 5856 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:09:32.883822 5856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4994c0392df844f7a85f2cf9bde9a904: Generated new TSK 0
I20260924 03:09:32.884788 5856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:09:32.908384 5845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:32.900787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43292 (local address 127.0.97.62:38819)
0924 03:09:32.901141 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:32.901148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:32.901284 (+ 136us) server_negotiation.cc:407] Connection header received
0924 03:09:32.901593 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:32.901597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:32.901677 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:32.901886 (+ 209us) server_negotiation.cc:226] Negotiated authn=JWT
0924 03:09:32.902385 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:32.903128 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:32.904664 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:32.905122 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:32.905277 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:32.905827 (+ 550us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 03:09:32.907715 (+ 1888us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44001/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 03:09:32.908073 (+ 358us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43292: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44001/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":93}
W20260924 03:09:32.908556 5845 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43292: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44001/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 03:09:32.908972 5871 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:32.900459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38819 (local address 127.0.0.1:43292)
0924 03:09:32.901151 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:32.901172 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:32.901377 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:32.901900 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:32.901905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:32.901917 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0924 03:09:32.902222 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:32.902230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:32.903291 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:32.903297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:32.904450 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:32.904462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:32.905265 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:32.905356 (+ 91us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:32.905500 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:32.905536 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 03:09:32.907932 (+ 2396us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44001/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 03:09:32.908180 (+ 248us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:38819: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44001/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":348,"threads_started":1}
I20260924 03:09:32.910772 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 5776
I20260924 03:09:32.911921 5837 generic_service.cc:196] Checking for leaks (request via RPC)
E20260924 03:09:32.921114 5872 webserver.cc:533] Webserver: sslize failed:
E20260924 03:09:32.921252 5872 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 03:09:33.004170 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 5776
2026-09-24T03:09:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (729 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T03:09:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:33Z Disabled control of system clock
I20260924 03:09:33.086990 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40545
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40545
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219327163871-388-0/testchainca.cert with env {}
W20260924 03:09:33.432857 5877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:33.433208 5877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:33.433307 5877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:33.441915 5877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:33.442072 5877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:33.442114 5877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:33.442173 5877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:33.442221 5877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 03:09:33.442281 5877 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 03:09:33.442317 5877 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 03:09:33.453331 5877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40545
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219327163871-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:33.454669 5877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:33.455941 5877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:33.468683 5885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:33.468654 5882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:33.468901 5883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:33.469516 5877 server_base.cc:1034] running on GCE node
I20260924 03:09:33.470412 5877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:33.472131 5877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:33.473414 5877 hybrid_clock.cc:648] HybridClock initialized: now 1790219373473361 us; error 75 us; skew 500 ppm
I20260924 03:09:33.473815 5877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:09:33.476639 5877 webserver.cc:460] Webserver started at http://127.0.97.62:43109/ using document root <none> and password file <none>
I20260924 03:09:33.477362 5877 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:33.477499 5877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:33.477818 5877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:33.480875 5877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "61081a2380964c02b29f2127c76350b1"
format_stamp: "Formatted at 2026-09-24 03:09:33 on dist-test-slave-5l98"
server_key: "f708b91b15f62ebdf6b9d76d25bdc48a"
server_key_iv: "ee3347963654b6379cc11cf0d3a9dd2b"
server_key_version: "encryptionkey@0"
I20260924 03:09:33.481705 5877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "61081a2380964c02b29f2127c76350b1"
format_stamp: "Formatted at 2026-09-24 03:09:33 on dist-test-slave-5l98"
server_key: "f708b91b15f62ebdf6b9d76d25bdc48a"
server_key_iv: "ee3347963654b6379cc11cf0d3a9dd2b"
server_key_version: "encryptionkey@0"
I20260924 03:09:33.487237 5877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:09:33.491179 5891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:33.492899 5877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:09:33.493111 5877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "61081a2380964c02b29f2127c76350b1"
format_stamp: "Formatted at 2026-09-24 03:09:33 on dist-test-slave-5l98"
server_key: "f708b91b15f62ebdf6b9d76d25bdc48a"
server_key_iv: "ee3347963654b6379cc11cf0d3a9dd2b"
server_key_version: "encryptionkey@0"
I20260924 03:09:33.493342 5877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:33.512339 5877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:33.513247 5877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:33.513543 5877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:33.532976 5877 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40545
I20260924 03:09:33.532999 5942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40545 every 8 connection(s)
I20260924 03:09:33.534377 5877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:09:33.538301 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 5877
I20260924 03:09:33.538517 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:09:33.538852 5943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:33.539434 388 external_mini_cluster.cc:1428] Setting key dd2293313fdc0497dc93fd470f97eea0
I20260924 03:09:33.552959 5943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1: Bootstrap starting.
I20260924 03:09:33.553635 5946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:33.541557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35542 (local address 127.0.97.62:40545)
0924 03:09:33.542702 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:33.542714 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:33.542760 (+ 46us) server_negotiation.cc:407] Connection header received
0924 03:09:33.544430 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:33.544452 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:33.545026 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:33.545444 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:33.546084 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:33.547386 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:33.548767 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:33.549480 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:33.549692 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:33.551062 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:33.551070 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:33.551088 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:33.551701 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:33.551782 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:33.551804 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:33.551959 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:33.552178 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 03:09:33.552574 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":489,"threads_started":1}
I20260924 03:09:33.556885 5943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:33.558135 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:09:33.558187 5943 log.cc:824] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:33.561703 5943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1: No bootstrap required, opened a new log
I20260924 03:09:33.569608 5943 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61081a2380964c02b29f2127c76350b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40545 } }
I20260924 03:09:33.569962 5943 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:33.570041 5943 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61081a2380964c02b29f2127c76350b1, State: Initialized, Role: FOLLOWER
I20260924 03:09:33.570554 5943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61081a2380964c02b29f2127c76350b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40545 } }
I20260924 03:09:33.570752 5943 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:09:33.570848 5943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:09:33.570986 5943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:33.573320 5943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61081a2380964c02b29f2127c76350b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40545 } }
I20260924 03:09:33.573793 5943 leader_election.cc:302] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61081a2380964c02b29f2127c76350b1; no voters:
I20260924 03:09:33.574463 5943 leader_election.cc:288] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:09:33.574698 5948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:33.575484 5948 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1 [term 1 LEADER]: Becoming Leader. State: Replica: 61081a2380964c02b29f2127c76350b1, State: Running, Role: LEADER
I20260924 03:09:33.576033 5948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61081a2380964c02b29f2127c76350b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40545 } }
I20260924 03:09:33.576715 5943 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:09:33.579267 5949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61081a2380964c02b29f2127c76350b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61081a2380964c02b29f2127c76350b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40545 } } }
I20260924 03:09:33.579597 5949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:33.580471 5956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:09:33.581177 5950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61081a2380964c02b29f2127c76350b1. Latest consensus state: current_term: 1 leader_uuid: "61081a2380964c02b29f2127c76350b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61081a2380964c02b29f2127c76350b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40545 } } }
I20260924 03:09:33.581594 5950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:33.586825 5956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:09:33.596619 5956 catalog_manager.cc:1347] Generated new cluster ID: 1e644b30720f4febb3b820ac3d200bab
I20260924 03:09:33.596755 5956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:09:33.606227 5956 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:09:33.608074 5956 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:09:33.621731 5956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61081a2380964c02b29f2127c76350b1: Generated new TSK 0
I20260924 03:09:33.622892 5956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:09:33.703253 5946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:33.640413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35554 (local address 127.0.97.62:40545)
0924 03:09:33.640623 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:33.640630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:33.641196 (+ 566us) server_negotiation.cc:407] Connection header received
0924 03:09:33.641417 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:33.641422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:33.641522 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:33.641697 (+ 175us) server_negotiation.cc:226] Negotiated authn=JWT
0924 03:09:33.642098 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:33.643386 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:33.644914 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:33.645605 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:33.645740 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:33.646324 (+ 584us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 03:09:33.702468 (+ 56144us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 03:09:33.702919 (+ 451us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35554: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":92}
W20260924 03:09:33.703471 5972 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:33.640252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40545 (local address 127.0.0.1:35554)
0924 03:09:33.641076 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:33.641092 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:33.641226 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:33.641714 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:33.641718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:33.641727 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 03:09:33.641957 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:33.641965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:33.643547 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:33.643551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:33.644767 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:33.644779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:33.645767 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:33.645826 (+ 59us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:33.645979 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:33.646022 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 03:09:33.702779 (+ 56757us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 03:09:33.703080 (+ 301us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:40545: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":508,"threads_started":1}
W20260924 03:09:33.703507 5946 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35554: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260924 03:09:33.704991 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 5877
I20260924 03:09:33.706291 5938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:33.810621 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 5877
2026-09-24T03:09:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (786 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (441 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1519 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:09:35 dist-test-slave-5l98 krb5kdc[5988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:09:35 dist-test-slave-5l98 krb5kdc[5988](info): set up 2 sockets
Sep 24 03:09:35 dist-test-slave-5l98 krb5kdc[5988](info): commencing operation
krb5kdc: starting...
W20260924 03:09:36.933821 388 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:09:37 dist-test-slave-5l98 krb5kdc[5988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:09:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:09:37.161362 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46823
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:37.489832 6001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:37.490139 6001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:37.490293 6001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:37.497602 6001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:37.497716 6001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:37.497823 6001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:37.497925 6001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:37.498008 6001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:09:37.507982 6001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:37.509497 6001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:37.510818 6001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:37.523149 6006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:37.523489 6009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:37.523604 6007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:37.525117 6001 server_base.cc:1034] running on GCE node
I20260924 03:09:37.525918 6001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:37.527459 6001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:37.528699 6001 hybrid_clock.cc:648] HybridClock initialized: now 1790219377528659 us; error 63 us; skew 500 ppm
I20260924 03:09:37.529129 6001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:37 dist-test-slave-5l98 krb5kdc[5988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:37.532842 6001 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:09:37.534502 6001 webserver.cc:460] Webserver started at http://127.0.97.62:34827/ using document root <none> and password file <none>
I20260924 03:09:37.535211 6001 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:37.535316 6001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:37.535583 6001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:37.538756 6001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "75fbdf18958d4e489a1d1356b8a61531"
format_stamp: "Formatted at 2026-09-24 03:09:37 on dist-test-slave-5l98"
server_key: "83bd1d7468dd487d4f96d64b5db16b6b"
server_key_iv: "4182f28392b4ba108b9ba40ca5826399"
server_key_version: "encryptionkey@0"
I20260924 03:09:37.539605 6001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "75fbdf18958d4e489a1d1356b8a61531"
format_stamp: "Formatted at 2026-09-24 03:09:37 on dist-test-slave-5l98"
server_key: "83bd1d7468dd487d4f96d64b5db16b6b"
server_key_iv: "4182f28392b4ba108b9ba40ca5826399"
server_key_version: "encryptionkey@0"
I20260924 03:09:37.545063 6001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:09:37.548884 6016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:37.550680 6001 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:09:37.550848 6001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "75fbdf18958d4e489a1d1356b8a61531"
format_stamp: "Formatted at 2026-09-24 03:09:37 on dist-test-slave-5l98"
server_key: "83bd1d7468dd487d4f96d64b5db16b6b"
server_key_iv: "4182f28392b4ba108b9ba40ca5826399"
server_key_version: "encryptionkey@0"
I20260924 03:09:37.551043 6001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:37.568681 6001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:37.573405 6001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:37.573659 6001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:37.593340 6001 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46823
I20260924 03:09:37.593375 6067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46823 every 8 connection(s)
I20260924 03:09:37.594718 6001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:09:37.598660 6068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:37.603225 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 6001
I20260924 03:09:37.603544 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:09:37.604264 388 external_mini_cluster.cc:1428] Setting key a997375e42f7625765bcfc61779b4141
I20260924 03:09:37.610551 6068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531: Bootstrap starting.
I20260924 03:09:37.614408 6068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:37.615643 6068 log.cc:824] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:37.618572 6068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531: No bootstrap required, opened a new log
Sep 24 03:09:37 dist-test-slave-5l98 krb5kdc[5988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:37.626448 6068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fbdf18958d4e489a1d1356b8a61531" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46823 } }
I20260924 03:09:37.626863 6068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:37.626999 6068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75fbdf18958d4e489a1d1356b8a61531, State: Initialized, Role: FOLLOWER
I20260924 03:09:37.627671 6068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fbdf18958d4e489a1d1356b8a61531" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46823 } }
I20260924 03:09:37.627933 6068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:09:37.628093 6068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:09:37.628269 6068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:37.628659 6071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:37.606530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54870 (local address 127.0.97.62:46823)
0924 03:09:37.607501 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:37.607513 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:37.607560 (+ 47us) server_negotiation.cc:407] Connection header received
0924 03:09:37.609460 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:37.609487 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:37.609914 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:37.610263 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:37.612152 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:37.613583 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:37.615358 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:37.616167 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:37.616380 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:37.620478 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:37.620538 (+ 60us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:37.620563 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:37.620601 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:37.623872 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:37.624879 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:37.624888 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:37.624895 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:37.625092 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:37.625495 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:37.625500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:37.625504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:37.626086 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:37.626396 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:37.626863 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 03:09:37.627347 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":511,"threads_started":1}
I20260924 03:09:37.630754 6068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fbdf18958d4e489a1d1356b8a61531" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46823 } }
I20260924 03:09:37.631251 6068 leader_election.cc:302] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75fbdf18958d4e489a1d1356b8a61531; no voters:
I20260924 03:09:37.631995 6068 leader_election.cc:288] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:09:37.632221 6073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:37.633155 6073 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531 [term 1 LEADER]: Becoming Leader. State: Replica: 75fbdf18958d4e489a1d1356b8a61531, State: Running, Role: LEADER
I20260924 03:09:37.633666 6073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fbdf18958d4e489a1d1356b8a61531" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46823 } }
I20260924 03:09:37.634367 6068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:09:37.636950 6075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75fbdf18958d4e489a1d1356b8a61531. Latest consensus state: current_term: 1 leader_uuid: "75fbdf18958d4e489a1d1356b8a61531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fbdf18958d4e489a1d1356b8a61531" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46823 } } }
I20260924 03:09:37.636885 6074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75fbdf18958d4e489a1d1356b8a61531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fbdf18958d4e489a1d1356b8a61531" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46823 } } }
I20260924 03:09:37.637327 6075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:37.637338 6074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:37.639086 6078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:09:37.646665 6078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:09:37.656591 6078 catalog_manager.cc:1347] Generated new cluster ID: b2e12e06bbb44baea13f6808ffda6f5a
I20260924 03:09:37.656726 6078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:09:37.677286 6078 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:09:37.678702 6078 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:09:37.690321 6078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75fbdf18958d4e489a1d1356b8a61531: Generated new TSK 0
I20260924 03:09:37.691242 6078 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 03:09:37.761615 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46823
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:38.121196 6094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:38.121487 6094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:38.121582 6094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:38.129398 6094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:38.129516 6094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:38.129678 6094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:09:38.141475 6094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:38.142796 6094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:38.144191 6094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:38.158196 6102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:38.158208 6100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:38.158231 6099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:38.158957 6094 server_base.cc:1034] running on GCE node
I20260924 03:09:38.159678 6094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:38.160791 6094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:38.162261 6094 hybrid_clock.cc:648] HybridClock initialized: now 1790219378162192 us; error 61 us; skew 500 ppm
I20260924 03:09:38.162648 6094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:38 dist-test-slave-5l98 krb5kdc[5988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:38.166342 6094 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:09:38.168615 6094 webserver.cc:460] Webserver started at http://127.0.97.1:38085/ using document root <none> and password file <none>
I20260924 03:09:38.169430 6094 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:38.169539 6094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:38.169788 6094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:38.172887 6094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "7c61d75a529040ec8925013caa3e3edd"
format_stamp: "Formatted at 2026-09-24 03:09:38 on dist-test-slave-5l98"
server_key: "df6f13025c190586b2576a7c6a5d2133"
server_key_iv: "0f0e38d083f363705eb9f5d0d5571b3e"
server_key_version: "encryptionkey@0"
I20260924 03:09:38.173692 6094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7c61d75a529040ec8925013caa3e3edd"
format_stamp: "Formatted at 2026-09-24 03:09:38 on dist-test-slave-5l98"
server_key: "df6f13025c190586b2576a7c6a5d2133"
server_key_iv: "0f0e38d083f363705eb9f5d0d5571b3e"
server_key_version: "encryptionkey@0"
I20260924 03:09:38.179172 6094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:09:38.183035 6109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:38.184844 6094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 03:09:38.185145 6094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "7c61d75a529040ec8925013caa3e3edd"
format_stamp: "Formatted at 2026-09-24 03:09:38 on dist-test-slave-5l98"
server_key: "df6f13025c190586b2576a7c6a5d2133"
server_key_iv: "0f0e38d083f363705eb9f5d0d5571b3e"
server_key_version: "encryptionkey@0"
I20260924 03:09:38.185379 6094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:38.217434 6094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:38.222002 6094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:38.222304 6094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:38.223356 6094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:38.224812 6094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:38.224928 6094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:38.225072 6094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:38.225152 6094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:38.261950 6094 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43987
I20260924 03:09:38.262048 6221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43987 every 8 connection(s)
I20260924 03:09:38.263268 6094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:09:38.267418 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 6094
I20260924 03:09:38.267707 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:09:38.268567 388 external_mini_cluster.cc:1428] Setting key f545392876332fac987d405640770b19
Sep 24 03:09:38 dist-test-slave-5l98 krb5kdc[5988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:38.291867 6224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:38.266196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57047 (local address 127.0.97.62:46823)
0924 03:09:38.266691 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:38.266698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:38.267877 (+ 1179us) server_negotiation.cc:407] Connection header received
0924 03:09:38.270291 (+ 2414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:38.270298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:38.270369 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:38.270491 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:38.273699 (+ 3208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:38.274940 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:38.278298 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:38.279310 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:38.279489 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:38.283590 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:38.283599 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:38.283607 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:38.283655 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:38.288051 (+ 4396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:38.289286 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:38.289291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:38.289293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:38.289417 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:38.289899 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:38.289909 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:38.289912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:38.290305 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:38.290468 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:38.291244 (+ 776us) server_negotiation.cc:299] Negotiation successful
0924 03:09:38.291472 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":256,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 03:09:38.292964 6225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:38.266765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46823 (local address 127.0.97.1:57047)
0924 03:09:38.267684 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:38.267747 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:38.269978 (+ 2231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:38.271324 (+ 1346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:38.271360 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:38.272293 (+ 933us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:38.273336 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:38.273373 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:38.275288 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:38.275299 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:38.278029 (+ 2730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:38.278055 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:38.279639 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:38.279647 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:38.279874 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:38.280682 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:38.280708 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:38.283311 (+ 2603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:38.288270 (+ 4959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:38.288278 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:38.288295 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:38.289121 (+ 826us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:38.289542 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:38.289547 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:38.289551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:38.289770 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:38.290442 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:38.290451 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:38.290866 (+ 415us) client_negotiation.cc:769] Sending connection context
0924 03:09:38.291142 (+ 276us) client_negotiation.cc:240] Negotiation successful
0924 03:09:38.291716 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":374,"threads_started":1}
I20260924 03:09:38.295378 6222 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46823
I20260924 03:09:38.295755 6222 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 03:09:38.296687 6222 heartbeater.cc:511] Master 127.0.97.62:46823 requested a full tablet report, sending...
I20260924 03:09:38.299186 6033 ts_manager.cc:194] Registered new tserver with Master: 7c61d75a529040ec8925013caa3e3edd (127.0.97.1:43987)
I20260924 03:09:38.301419 6033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 03:09:38.329917 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46823
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:38.691752 6228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:38.692139 6228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:38.692324 6228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:38.701143 6228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:38.701316 6228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:38.701560 6228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:09:38.712839 6228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:38.714331 6228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:38.715770 6228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:38.730247 6233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:38.730217 6234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:38.730695 6236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:38.731446 6228 server_base.cc:1034] running on GCE node
I20260924 03:09:38.732113 6228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:38.733233 6228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:38.734586 6228 hybrid_clock.cc:648] HybridClock initialized: now 1790219378734545 us; error 96 us; skew 500 ppm
I20260924 03:09:38.735008 6228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:38 dist-test-slave-5l98 krb5kdc[5988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:38.738829 6228 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:09:38.740427 6228 webserver.cc:460] Webserver started at http://127.0.97.2:46557/ using document root <none> and password file <none>
I20260924 03:09:38.741272 6228 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:38.741410 6228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:38.741734 6228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:38.744992 6228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "3fdc5bf23a63494391c124e864a2d733"
format_stamp: "Formatted at 2026-09-24 03:09:38 on dist-test-slave-5l98"
server_key: "6badab0a842b1fd95bf49f72fd4ba644"
server_key_iv: "10d93c8a9ce98c21716fa26cec22874b"
server_key_version: "encryptionkey@0"
I20260924 03:09:38.745817 6228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3fdc5bf23a63494391c124e864a2d733"
format_stamp: "Formatted at 2026-09-24 03:09:38 on dist-test-slave-5l98"
server_key: "6badab0a842b1fd95bf49f72fd4ba644"
server_key_iv: "10d93c8a9ce98c21716fa26cec22874b"
server_key_version: "encryptionkey@0"
I20260924 03:09:38.752229 6228 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260924 03:09:38.756704 6243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:38.758721 6228 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:09:38.758932 6228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "3fdc5bf23a63494391c124e864a2d733"
format_stamp: "Formatted at 2026-09-24 03:09:38 on dist-test-slave-5l98"
server_key: "6badab0a842b1fd95bf49f72fd4ba644"
server_key_iv: "10d93c8a9ce98c21716fa26cec22874b"
server_key_version: "encryptionkey@0"
I20260924 03:09:38.759174 6228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:38.781517 6228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:38.786293 6228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:38.786610 6228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:38.787637 6228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:38.789216 6228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:38.789295 6228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:38.789448 6228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:38.789569 6228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:38.824682 6228 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39845
I20260924 03:09:38.824733 6355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39845 every 8 connection(s)
I20260924 03:09:38.825966 6228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:09:38.833365 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 6228
I20260924 03:09:38.833612 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:09:38.834116 388 external_mini_cluster.cc:1428] Setting key 41878120ae0135f371deb558d7618c6e
Sep 24 03:09:38 dist-test-slave-5l98 krb5kdc[5988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:38.848104 6358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:38.828765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46939 (local address 127.0.97.62:46823)
0924 03:09:38.829230 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:38.829235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:38.830438 (+ 1203us) server_negotiation.cc:407] Connection header received
0924 03:09:38.832333 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:38.832338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:38.832428 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:38.832572 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:38.834747 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:38.835603 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:38.837347 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:38.837768 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:38.837899 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:38.841815 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:38.841820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:38.841824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:38.841856 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:38.844816 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:38.845622 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:38.845628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:38.845632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:38.845765 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:38.846218 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:38.846223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:38.846227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:38.846497 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:38.846668 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:38.847443 (+ 775us) server_negotiation.cc:299] Negotiation successful
0924 03:09:38.847683 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":228,"threads_started":1}
I20260924 03:09:38.849931 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:38.829390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46823 (local address 127.0.97.2:46939)
0924 03:09:38.830203 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:38.830247 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:38.832054 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:38.832802 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:38.832810 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:38.833346 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:38.834407 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:38.834431 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:38.835751 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:38.835756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:38.837193 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:38.837206 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:38.837999 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:38.838007 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:38.838223 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:38.838917 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:38.838944 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:38.841474 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:38.844980 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:38.844990 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:38.845011 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:38.845472 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:38.845880 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:38.845886 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:38.845890 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:38.846095 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:38.846631 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:38.846640 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:38.847041 (+ 401us) client_negotiation.cc:769] Sending connection context
0924 03:09:38.847331 (+ 290us) client_negotiation.cc:240] Negotiation successful
0924 03:09:38.847928 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":363,"threads_started":1}
I20260924 03:09:38.851805 6356 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46823
I20260924 03:09:38.852154 6356 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:38.852978 6356 heartbeater.cc:511] Master 127.0.97.62:46823 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 03:09:38.854751 6033 ts_manager.cc:194] Registered new tserver with Master: 3fdc5bf23a63494391c124e864a2d733 (127.0.97.2:39845)
I20260924 03:09:38.855852 6033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 03:09:38.885763 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46823
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:39.235271 6362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:39.235571 6362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:39.235737 6362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:39.243307 6362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:39.243432 6362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:39.243610 6362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:09:39.253925 6362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:39.255483 6362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:39.256986 6362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:39.273568 6368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:39.273685 6367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:39.274175 6370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:39.274603 6362 server_base.cc:1034] running on GCE node
I20260924 03:09:39.275240 6362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:39.276242 6362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:39.277472 6362 hybrid_clock.cc:648] HybridClock initialized: now 1790219379277398 us; error 64 us; skew 500 ppm
I20260924 03:09:39.277833 6362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:39 dist-test-slave-5l98 krb5kdc[5988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:39.281630 6362 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:09:39.283560 6362 webserver.cc:460] Webserver started at http://127.0.97.3:37197/ using document root <none> and password file <none>
I20260924 03:09:39.284387 6362 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:39.284579 6362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:39.284901 6362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:39.288669 6362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "8294158e1f38405c8eb07e1482414742"
format_stamp: "Formatted at 2026-09-24 03:09:39 on dist-test-slave-5l98"
server_key: "bce32df4005d3586631262c6e5d5e13e"
server_key_iv: "ace34afbb3c151023817b59cd81711fe"
server_key_version: "encryptionkey@0"
I20260924 03:09:39.289644 6362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8294158e1f38405c8eb07e1482414742"
format_stamp: "Formatted at 2026-09-24 03:09:39 on dist-test-slave-5l98"
server_key: "bce32df4005d3586631262c6e5d5e13e"
server_key_iv: "ace34afbb3c151023817b59cd81711fe"
server_key_version: "encryptionkey@0"
I20260924 03:09:39.295014 6362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:09:39.298964 6377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:39.300345 6362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:09:39.300622 6362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "8294158e1f38405c8eb07e1482414742"
format_stamp: "Formatted at 2026-09-24 03:09:39 on dist-test-slave-5l98"
server_key: "bce32df4005d3586631262c6e5d5e13e"
server_key_iv: "ace34afbb3c151023817b59cd81711fe"
server_key_version: "encryptionkey@0"
I20260924 03:09:39.300840 6362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:39.305696 6222 heartbeater.cc:503] Master 127.0.97.62:46823 was elected leader, sending a full tablet report...
I20260924 03:09:39.330199 6362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:39.335456 6362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:39.335734 6362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:39.336795 6362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:39.338301 6362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:39.338424 6362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:39.338586 6362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:39.338661 6362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:39.377869 6362 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38971
I20260924 03:09:39.377910 6489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38971 every 8 connection(s)
I20260924 03:09:39.379320 6362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:09:39.380781 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 6362
I20260924 03:09:39.380950 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:09:39.381538 388 external_mini_cluster.cc:1428] Setting key 96c907de2a771fac493848eccfffcb14
Sep 24 03:09:39 dist-test-slave-5l98 krb5kdc[5988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:09:39.403499 6492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:39.382603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40217 (local address 127.0.97.62:46823)
0924 03:09:39.383211 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:39.383216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:39.384614 (+ 1398us) server_negotiation.cc:407] Connection header received
0924 03:09:39.386942 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:39.386947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:39.387034 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:39.387130 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:39.389329 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:39.390294 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:39.392065 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:39.392457 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:39.392651 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:39.397236 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:39.397242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:39.397246 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:39.397290 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:39.400363 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:39.401343 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:39.401347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:39.401349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:39.401520 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:39.401941 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:39.401944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:39.401946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:39.402208 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:39.402316 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:39.403028 (+ 712us) server_negotiation.cc:299] Negotiation successful
0924 03:09:39.403216 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":258,"threads_started":1}
I20260924 03:09:39.404819 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:39.383289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46823 (local address 127.0.97.3:40217)
0924 03:09:39.384411 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:39.384465 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:39.386633 (+ 2168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:39.387349 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:39.387358 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:39.388236 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:39.389094 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:39.389122 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:39.390455 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:39.390459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:39.391849 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:39.391862 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:39.392781 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:39.392786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:39.393035 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:39.393925 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:39.393950 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:39.396894 (+ 2944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:39.400552 (+ 3658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:39.400565 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:39.400589 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:39.401197 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:39.401625 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:39.401629 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:39.401631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:39.401822 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:39.402323 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:39.402330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:39.402678 (+ 348us) client_negotiation.cc:769] Sending connection context
0924 03:09:39.402902 (+ 224us) client_negotiation.cc:240] Negotiation successful
0924 03:09:39.403320 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":398,"threads_started":1}
I20260924 03:09:39.406666 6490 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46823
I20260924 03:09:39.406947 6490 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:39.407701 6490 heartbeater.cc:511] Master 127.0.97.62:46823 requested a full tablet report, sending...
I20260924 03:09:39.409473 6033 ts_manager.cc:194] Registered new tserver with Master: 8294158e1f38405c8eb07e1482414742 (127.0.97.3:38971)
I20260924 03:09:39.410307 6033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40217
I20260924 03:09:39.411231 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/test-admin.keytab.
Sep 24 03:09:39 dist-test-slave-5l98 krb5kdc[5988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:39.440042 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 03:09:39.445714 6498 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:39.443014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33552 (local address 127.0.97.1:43987)
0924 03:09:39.443726 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:39.443738 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:39.443771 (+ 33us) server_negotiation.cc:407] Connection header received
0924 03:09:39.443845 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:39.443854 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:39.444095 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:39.444269 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:39.445265 (+ 996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33552: BlockingRecv error: recv got EOF from 127.0.0.1:33552 (error 108)
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":384,"threads_started":1}
W20260924 03:09:39.445906 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/krb5cc' of type FILE
Sep 24 03:09:39 dist-test-slave-5l98 krb5kdc[5988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:39.447678 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 03:09:39 dist-test-slave-5l98 krb5kdc[5988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:39.449438 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 03:09:39 dist-test-slave-5l98 krb5kdc[5988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:09:39.466779 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:39.451392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33556 (local address 127.0.97.1:43987)
0924 03:09:39.451581 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:39.451590 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:39.451633 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:09:39.451860 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:39.451870 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:39.451936 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:39.452046 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:39.453233 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:39.454301 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:39.455642 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:39.456295 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:39.456460 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:39.460598 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:39.460613 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:39.460628 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:39.460694 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:39.463648 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:39.464576 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:39.464586 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:39.464590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:39.464734 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:39.465133 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:39.465140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:39.465143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:39.465587 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:39.465896 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:39.466209 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 03:09:39.466483 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 03:09:39.473201 6498 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:39.471299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.0.97.1:43987)
0924 03:09:39.471550 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:39.471554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:39.471756 (+ 202us) server_negotiation.cc:407] Connection header received
0924 03:09:39.471906 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:39.471909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:39.472019 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:39.472104 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:39.472919 (+ 815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33560: BlockingRecv error: recv got EOF from 127.0.0.1:33560 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 03:09:39.473838 388 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/krb5cc)
Sep 24 03:09:39 dist-test-slave-5l98 krb5kdc[5988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:39.475448 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 03:09:39 dist-test-slave-5l98 krb5kdc[5988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:39.476881 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 03:09:39 dist-test-slave-5l98 krb5kdc[5988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:09:39.492022 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:39.478536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33574 (local address 127.0.97.1:43987)
0924 03:09:39.478692 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:39.478697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:39.478732 (+ 35us) server_negotiation.cc:407] Connection header received
0924 03:09:39.478886 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:39.478890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:39.478942 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:39.479100 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:39.480155 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:39.480947 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:39.482165 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:39.482580 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:39.482674 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:39.486501 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:39.486506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:39.486509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:39.486543 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:39.489429 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:39.490173 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:39.490177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:39.490180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:39.490307 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:39.490711 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:39.490714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:39.490716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:39.491006 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:39.491211 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:39.491522 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 03:09:39.491685 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 03:09:39.496862 6498 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:39.495072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33586 (local address 127.0.97.1:43987)
0924 03:09:39.495289 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:39.495295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:39.495515 (+ 220us) server_negotiation.cc:407] Connection header received
0924 03:09:39.495685 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:39.495688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:39.495766 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:39.495872 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:39.496604 (+ 732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33586: BlockingRecv error: recv got EOF from 127.0.0.1:33586 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260924 03:09:39.497094 388 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219327163871-388-0/krb5kdc/krb5cc)
Sep 24 03:09:39 dist-test-slave-5l98 krb5kdc[5988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:39.498775 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 03:09:39 dist-test-slave-5l98 krb5kdc[5988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:39.500237 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 03:09:39 dist-test-slave-5l98 krb5kdc[5988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:09:39.515661 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:39.501690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.0.97.1:43987)
0924 03:09:39.501896 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:39.501904 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:39.502178 (+ 274us) server_negotiation.cc:407] Connection header received
0924 03:09:39.502372 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:39.502377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:39.502464 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:39.502586 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:39.503503 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:39.504229 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:39.505577 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:39.506007 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:39.506140 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:39.510143 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:39.510161 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:39.510164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:39.510200 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:39.513060 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:39.513727 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:39.513738 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:39.513741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:39.513900 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:39.514263 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:39.514270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:39.514273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:39.514545 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:39.514911 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:39.515070 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 03:09:39.515291 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 03:09:39.518090 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 6094
I20260924 03:09:39.531282 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:39.518953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.0.97.1:43987)
0924 03:09:39.519132 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:39.519137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:39.519656 (+ 519us) server_negotiation.cc:407] Connection header received
0924 03:09:39.519796 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:39.519799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:39.519863 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:39.519961 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:39.521227 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:39.521949 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:39.523286 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:39.523789 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:39.523895 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:39.526249 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:39.526254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:39.526257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:39.526290 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:39.528978 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:39.529559 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:39.529563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:39.529566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:39.529690 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:39.530082 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:39.530086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:39.530088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:39.530344 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:39.530566 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:39.530846 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 03:09:39.531015 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:09:39.531919 6217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:39.617760 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 6094
I20260924 03:09:39.634166 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 6228
Sep 24 03:09:39 dist-test-slave-5l98 krb5kdc[5988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 03:09:39.650799 6511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:39.635182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44404 (local address 127.0.97.2:39845)
0924 03:09:39.635773 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:39.635780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:39.635806 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:09:39.635871 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:39.635877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:39.636086 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:39.636217 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:39.637264 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:39.638326 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:39.639660 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:39.640278 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:39.640385 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:39.644841 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:39.644854 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:39.644868 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:39.644909 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:39.647914 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:39.648793 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:39.648801 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:39.648816 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:39.648946 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:39.649344 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:39.649347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:39.649349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:39.649718 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:39.649946 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:39.650231 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 03:09:39.650428 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":298,"threads_started":1}
I20260924 03:09:39.652129 6351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:39.739068 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 6228
I20260924 03:09:39.755251 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 6362
Sep 24 03:09:39 dist-test-slave-5l98 krb5kdc[5988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 03:09:39.772768 6513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:39.756442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47364 (local address 127.0.97.3:38971)
0924 03:09:39.757243 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:39.757251 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:39.757280 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:09:39.757338 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:39.757346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:39.757577 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:39.757712 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:39.758695 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:39.759851 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:39.761466 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:39.762057 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:39.762167 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:39.766336 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:39.766349 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:39.766364 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:39.766400 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:39.769526 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:39.770489 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:39.770497 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:39.770499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:39.770617 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:39.771135 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:39.771139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:39.771141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:39.771511 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:39.771766 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:39.772159 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 03:09:39.772371 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":479,"threads_started":1}
I20260924 03:09:39.774158 6485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:39.859086 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 6362
I20260924 03:09:39.874784 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 6001
I20260924 03:09:39.876189 6063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:39.981477 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 6001
2026-09-24T03:09:39Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4216 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:09:40 dist-test-slave-5l98 krb5kdc[6518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:09:40 dist-test-slave-5l98 krb5kdc[6518](info): set up 2 sockets
Sep 24 03:09:40 dist-test-slave-5l98 krb5kdc[6518](info): commencing operation
krb5kdc: starting...
W20260924 03:09:41.161523 388 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:09:41 dist-test-slave-5l98 krb5kdc[6518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:09:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260924 03:09:41.401947 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34603
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:41.737758 6531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:41.738097 6531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:41.738250 6531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:41.746110 6531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:41.746236 6531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:41.746317 6531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:41.746402 6531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:41.746687 6531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:09:41.757915 6531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:41.759598 6531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:41.761159 6531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:41.774260 6536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:41.774356 6539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:41.774328 6537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:41.775930 6531 server_base.cc:1034] running on GCE node
I20260924 03:09:41.776964 6531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:41.778581 6531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:41.779847 6531 hybrid_clock.cc:648] HybridClock initialized: now 1790219381779783 us; error 56 us; skew 500 ppm
I20260924 03:09:41.780340 6531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:41 dist-test-slave-5l98 krb5kdc[6518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219381, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:41.791440 6531 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260924 03:09:41.793301 6531 webserver.cc:460] Webserver started at http://127.0.97.62:39837/ using document root <none> and password file <none>
I20260924 03:09:41.794039 6531 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:41.794193 6531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:41.794566 6531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:41.797741 6531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "0a226d7c36f749c78880160588ba0ba1"
format_stamp: "Formatted at 2026-09-24 03:09:41 on dist-test-slave-5l98"
server_key: "0825958a7f8d61c397c05b3ca4fcf3f3"
server_key_iv: "488881f6cb7a19e2bacdf8e63f65cd4d"
server_key_version: "encryptionkey@0"
I20260924 03:09:41.798543 6531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "0a226d7c36f749c78880160588ba0ba1"
format_stamp: "Formatted at 2026-09-24 03:09:41 on dist-test-slave-5l98"
server_key: "0825958a7f8d61c397c05b3ca4fcf3f3"
server_key_iv: "488881f6cb7a19e2bacdf8e63f65cd4d"
server_key_version: "encryptionkey@0"
I20260924 03:09:41.804145 6531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:09:41.807901 6546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:41.809684 6531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:09:41.809890 6531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "0a226d7c36f749c78880160588ba0ba1"
format_stamp: "Formatted at 2026-09-24 03:09:41 on dist-test-slave-5l98"
server_key: "0825958a7f8d61c397c05b3ca4fcf3f3"
server_key_iv: "488881f6cb7a19e2bacdf8e63f65cd4d"
server_key_version: "encryptionkey@0"
I20260924 03:09:41.810194 6531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:41.828791 6531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:41.833786 6531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:41.834136 6531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:41.854279 6531 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34603
I20260924 03:09:41.854291 6597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34603 every 8 connection(s)
I20260924 03:09:41.855792 6531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:09:41.860167 6598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:41.865968 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 6531
I20260924 03:09:41.866212 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:09:41.867058 388 external_mini_cluster.cc:1428] Setting key 220fbfa055a74be9bdea71168ed6d9d9
I20260924 03:09:41.873457 6598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1: Bootstrap starting.
I20260924 03:09:41.877795 6598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:41.879169 6598 log.cc:824] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:41.883376 6598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1: No bootstrap required, opened a new log
Sep 24 03:09:41 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 03:09:41.892194 6598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } }
I20260924 03:09:41.892699 6598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:41.892819 6598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a226d7c36f749c78880160588ba0ba1, State: Initialized, Role: FOLLOWER
I20260924 03:09:41.893445 6598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } }
I20260924 03:09:41.893712 6598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:09:41.893855 6598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:09:41.894078 6598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:41.896281 6601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:41.869440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55636 (local address 127.0.97.62:34603)
0924 03:09:41.870322 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:41.870335 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:41.870396 (+ 61us) server_negotiation.cc:407] Connection header received
0924 03:09:41.872828 (+ 2432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:41.872861 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:41.873404 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:41.873845 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:41.875890 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:41.877746 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:41.879660 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:41.881037 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:41.881198 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:41.886285 (+ 5087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:41.886294 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:41.886314 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:41.886360 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:41.890470 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:41.891706 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:41.891717 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:41.891724 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:41.891914 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:41.892617 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:41.892623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:41.892627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:41.893425 (+ 798us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:41.893737 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:41.894242 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 03:09:41.894806 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":413,"threads_started":1}
I20260924 03:09:41.897370 6598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } }
I20260924 03:09:41.898119 6598 leader_election.cc:302] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a226d7c36f749c78880160588ba0ba1; no voters:
I20260924 03:09:41.899227 6598 leader_election.cc:288] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:09:41.899369 6603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:41.901095 6603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [term 1 LEADER]: Becoming Leader. State: Replica: 0a226d7c36f749c78880160588ba0ba1, State: Running, Role: LEADER
I20260924 03:09:41.901618 6603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } }
I20260924 03:09:41.902011 6598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:09:41.906699 6605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a226d7c36f749c78880160588ba0ba1. Latest consensus state: current_term: 1 leader_uuid: "0a226d7c36f749c78880160588ba0ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } }
I20260924 03:09:41.907709 6605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:41.907339 6604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a226d7c36f749c78880160588ba0ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } }
I20260924 03:09:41.907842 6604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:41.913285 6613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:09:41.918678 6613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:09:41.927492 6613 catalog_manager.cc:1347] Generated new cluster ID: adc5298acf804509a20dfe0c10861d58
I20260924 03:09:41.927634 6613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:09:41.936946 6613 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:09:41.938597 6613 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:09:41.950726 6613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1: Generated new TSK 0
I20260924 03:09:41.951682 6613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260924 03:09:42.022523 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34603
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:42.377697 6624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:42.378021 6624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:42.378185 6624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:42.385818 6624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:42.385953 6624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:42.386140 6624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:09:42.397063 6624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:42.398432 6624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:42.399755 6624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:42.415890 6629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:42.415774 6630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:42.416121 6632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:42.417126 6624 server_base.cc:1034] running on GCE node
I20260924 03:09:42.417801 6624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:42.418833 6624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:42.420136 6624 hybrid_clock.cc:648] HybridClock initialized: now 1790219382420081 us; error 54 us; skew 500 ppm
I20260924 03:09:42.420578 6624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:42 dist-test-slave-5l98 krb5kdc[6518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219382, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:42.424177 6624 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260924 03:09:42.425979 6624 webserver.cc:460] Webserver started at http://127.0.97.1:45307/ using document root <none> and password file <none>
I20260924 03:09:42.426730 6624 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:42.426898 6624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:42.427222 6624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:42.430500 6624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "00cb9dff7a9b430c9ad5224aa776edee"
format_stamp: "Formatted at 2026-09-24 03:09:42 on dist-test-slave-5l98"
server_key: "ee60c7f56658d106704940694ece96b9"
server_key_iv: "c731830e7a501ca7819487702e49d475"
server_key_version: "encryptionkey@0"
I20260924 03:09:42.431356 6624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "00cb9dff7a9b430c9ad5224aa776edee"
format_stamp: "Formatted at 2026-09-24 03:09:42 on dist-test-slave-5l98"
server_key: "ee60c7f56658d106704940694ece96b9"
server_key_iv: "c731830e7a501ca7819487702e49d475"
server_key_version: "encryptionkey@0"
I20260924 03:09:42.437103 6624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 03:09:42.441205 6639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:42.442901 6624 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:09:42.443123 6624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "00cb9dff7a9b430c9ad5224aa776edee"
format_stamp: "Formatted at 2026-09-24 03:09:42 on dist-test-slave-5l98"
server_key: "ee60c7f56658d106704940694ece96b9"
server_key_iv: "c731830e7a501ca7819487702e49d475"
server_key_version: "encryptionkey@0"
I20260924 03:09:42.443358 6624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:42.472787 6624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:42.478366 6624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:42.478751 6624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:42.479817 6624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:42.481345 6624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:42.481454 6624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:42.481617 6624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:42.481693 6624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:42.517604 6624 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37927
I20260924 03:09:42.517730 6751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37927 every 8 connection(s)
I20260924 03:09:42.519057 6624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:09:42.528355 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 6624
I20260924 03:09:42.528625 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:09:42.529307 388 external_mini_cluster.cc:1428] Setting key c44aeddf4c72fb2c5a636a4364e4bc93
Sep 24 03:09:42 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219382, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 03:09:42.541051 6754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:42.521822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56457 (local address 127.0.97.62:34603)
0924 03:09:42.522271 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:42.522277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:42.523209 (+ 932us) server_negotiation.cc:407] Connection header received
0924 03:09:42.525193 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:42.525197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:42.525269 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:42.525367 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:42.527062 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:42.527797 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:42.529863 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:42.530264 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:42.530364 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:42.534109 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:42.534114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:42.534118 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:42.534154 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:42.537622 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:42.538602 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:42.538608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:42.538616 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:42.538743 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:42.539217 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:42.539227 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:42.539230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:42.539514 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:42.539671 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:42.540407 (+ 736us) server_negotiation.cc:299] Negotiation successful
0924 03:09:42.540680 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":223,"threads_started":1}
I20260924 03:09:42.542039 6755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:42.522304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.1:56457)
0924 03:09:42.523029 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:42.523073 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:42.524861 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:42.525727 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:42.525736 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:42.526295 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:42.526881 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:42.526899 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:42.528148 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:42.528152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:42.529683 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:42.529701 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:42.530499 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:42.530509 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:42.530723 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:42.531499 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:42.531525 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:42.533844 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:42.537832 (+ 3988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:42.537843 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:42.537868 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:42.538444 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:42.538867 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:42.538872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:42.538875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:42.539078 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:42.539653 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:42.539662 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:42.540026 (+ 364us) client_negotiation.cc:769] Sending connection context
0924 03:09:42.540298 (+ 272us) client_negotiation.cc:240] Negotiation successful
0924 03:09:42.540862 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":318,"threads_started":1}
I20260924 03:09:42.543972 6752 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34603
I20260924 03:09:42.544415 6752 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:42.545363 6752 heartbeater.cc:511] Master 127.0.97.62:34603 requested a full tablet report, sending...
I20260924 03:09:42.548323 6563 ts_manager.cc:194] Registered new tserver with Master: 00cb9dff7a9b430c9ad5224aa776edee (127.0.97.1:37927)
I20260924 03:09:42.550757 6563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56457
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260924 03:09:42.586908 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34603
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:42.944131 6758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:42.944446 6758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:42.944636 6758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:42.952317 6758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:42.952448 6758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:42.952657 6758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:09:42.963564 6758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:42.965154 6758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:42.966572 6758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:42.983110 6763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:42.983321 6766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:42.983026 6764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:42.983052 6758 server_base.cc:1034] running on GCE node
I20260924 03:09:42.984426 6758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:42.985790 6758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:42.987059 6758 hybrid_clock.cc:648] HybridClock initialized: now 1790219382987021 us; error 41 us; skew 500 ppm
I20260924 03:09:42.987422 6758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:42 dist-test-slave-5l98 krb5kdc[6518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219382, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:42.991099 6758 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260924 03:09:42.992900 6758 webserver.cc:460] Webserver started at http://127.0.97.2:38691/ using document root <none> and password file <none>
I20260924 03:09:42.993650 6758 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:42.993783 6758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:42.994143 6758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:42.997468 6758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "765e21fdfb6049d48dcb457debd0e02f"
format_stamp: "Formatted at 2026-09-24 03:09:42 on dist-test-slave-5l98"
server_key: "b1f414bba8013fbc1877d965977533c1"
server_key_iv: "f12b47d6f0f3445c4c047c1311c8d4ea"
server_key_version: "encryptionkey@0"
I20260924 03:09:42.998296 6758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "765e21fdfb6049d48dcb457debd0e02f"
format_stamp: "Formatted at 2026-09-24 03:09:42 on dist-test-slave-5l98"
server_key: "b1f414bba8013fbc1877d965977533c1"
server_key_iv: "f12b47d6f0f3445c4c047c1311c8d4ea"
server_key_version: "encryptionkey@0"
I20260924 03:09:43.004206 6758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:09:43.009650 6773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:43.012101 6758 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260924 03:09:43.012358 6758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "765e21fdfb6049d48dcb457debd0e02f"
format_stamp: "Formatted at 2026-09-24 03:09:42 on dist-test-slave-5l98"
server_key: "b1f414bba8013fbc1877d965977533c1"
server_key_iv: "f12b47d6f0f3445c4c047c1311c8d4ea"
server_key_version: "encryptionkey@0"
I20260924 03:09:43.012682 6758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:43.040792 6758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:43.045993 6758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:43.046352 6758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:43.047648 6758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:43.049289 6758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:43.049414 6758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:43.049583 6758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:43.049664 6758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:43.090397 6758 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43547
I20260924 03:09:43.090459 6885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43547 every 8 connection(s)
I20260924 03:09:43.091706 6758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:09:43.101547 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 6758
I20260924 03:09:43.101734 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:09:43.102311 388 external_mini_cluster.cc:1428] Setting key 9bde3e91822b1596325df34fbd5f19eb
Sep 24 03:09:43 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219382, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 03:09:43.116230 6888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:43.094576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57439 (local address 127.0.97.62:34603)
0924 03:09:43.095031 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:43.095037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:43.095905 (+ 868us) server_negotiation.cc:407] Connection header received
0924 03:09:43.098087 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:43.098093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:43.098185 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:43.098304 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:43.099853 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:43.100657 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:43.103054 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:43.103651 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:43.103788 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:43.109240 (+ 5452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:43.109247 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:43.109251 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:43.109301 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:43.112730 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:43.113663 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:43.113669 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:43.113672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:43.113815 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:43.114318 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:43.114325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:43.114328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:43.114607 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:43.114771 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:43.115590 (+ 819us) server_negotiation.cc:299] Negotiation successful
0924 03:09:43.115825 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":344,"thread_start_us":242,"threads_started":1}
I20260924 03:09:43.117484 6889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:43.095009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.2:57439)
0924 03:09:43.095738 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:43.095776 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:43.097785 (+ 2009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:43.098542 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:43.098551 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:43.099063 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:43.099661 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:43.099679 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:43.100807 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:43.100817 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:43.102864 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:43.102881 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:43.103886 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:43.103895 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:43.104145 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:43.105147 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:43.105186 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:43.108849 (+ 3663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:43.112948 (+ 4099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:43.112960 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:43.112983 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:43.113515 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:43.113938 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:43.113943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:43.113947 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:43.114199 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:43.114744 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:43.114753 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:43.115218 (+ 465us) client_negotiation.cc:769] Sending connection context
0924 03:09:43.115513 (+ 295us) client_negotiation.cc:240] Negotiation successful
0924 03:09:43.116080 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":272,"threads_started":1}
I20260924 03:09:43.119149 6886 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34603
I20260924 03:09:43.119530 6886 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:43.120476 6886 heartbeater.cc:511] Master 127.0.97.62:34603 requested a full tablet report, sending...
I20260924 03:09:43.122438 6563 ts_manager.cc:194] Registered new tserver with Master: 765e21fdfb6049d48dcb457debd0e02f (127.0.97.2:43547)
I20260924 03:09:43.123812 6563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57439
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260924 03:09:43.156908 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34603
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:43.492465 6892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:43.492771 6892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:43.492959 6892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:43.500299 6892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:43.500417 6892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:43.500641 6892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:09:43.511438 6892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:43.512876 6892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:43.514318 6892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:43.529181 6897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:43.529198 6900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:43.529371 6898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:43.530136 6892 server_base.cc:1034] running on GCE node
I20260924 03:09:43.530704 6892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:43.531620 6892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:43.532827 6892 hybrid_clock.cc:648] HybridClock initialized: now 1790219383532781 us; error 49 us; skew 500 ppm
I20260924 03:09:43.533203 6892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:43 dist-test-slave-5l98 krb5kdc[6518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:43.536784 6892 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260924 03:09:43.538443 6892 webserver.cc:460] Webserver started at http://127.0.97.3:42127/ using document root <none> and password file <none>
I20260924 03:09:43.539144 6892 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:43.539294 6892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:43.539628 6892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:43.542665 6892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "8a3d53cd8e5c47cbb4da460f4198db40"
format_stamp: "Formatted at 2026-09-24 03:09:43 on dist-test-slave-5l98"
server_key: "4fa4215cb93da13cf5a04302add310d7"
server_key_iv: "3f36a38cb34ee20cd6d73bbb5d3dd44e"
server_key_version: "encryptionkey@0"
I20260924 03:09:43.543474 6892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8a3d53cd8e5c47cbb4da460f4198db40"
format_stamp: "Formatted at 2026-09-24 03:09:43 on dist-test-slave-5l98"
server_key: "4fa4215cb93da13cf5a04302add310d7"
server_key_iv: "3f36a38cb34ee20cd6d73bbb5d3dd44e"
server_key_version: "encryptionkey@0"
I20260924 03:09:43.549185 6892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:09:43.553735 6907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:43.554459 6752 heartbeater.cc:503] Master 127.0.97.62:34603 was elected leader, sending a full tablet report...
I20260924 03:09:43.555596 6892 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260924 03:09:43.555781 6892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "8a3d53cd8e5c47cbb4da460f4198db40"
format_stamp: "Formatted at 2026-09-24 03:09:43 on dist-test-slave-5l98"
server_key: "4fa4215cb93da13cf5a04302add310d7"
server_key_iv: "3f36a38cb34ee20cd6d73bbb5d3dd44e"
server_key_version: "encryptionkey@0"
I20260924 03:09:43.555992 6892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:43.570955 6892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:43.575716 6892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:43.576042 6892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:43.577135 6892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:43.578606 6892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:43.578679 6892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:43.578835 6892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:43.578912 6892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:43.614189 6892 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37999
I20260924 03:09:43.614287 7019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37999 every 8 connection(s)
I20260924 03:09:43.615655 6892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:09:43.619611 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 6892
I20260924 03:09:43.619831 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:09:43.620376 388 external_mini_cluster.cc:1428] Setting key 658e0b7693178b16df8a692887f93afd
Sep 24 03:09:43 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 03:09:43.636902 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:43.618860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38743 (local address 127.0.97.62:34603)
0924 03:09:43.619397 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:43.619401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:43.620390 (+ 989us) server_negotiation.cc:407] Connection header received
0924 03:09:43.622574 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:43.622580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:43.622671 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:43.622790 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:43.624536 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:43.625319 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:43.626914 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:43.627317 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:43.627453 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:43.631072 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:43.631078 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:43.631081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:43.631115 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:43.633743 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:43.634494 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:43.634501 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:43.634504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:43.634647 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:43.635078 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:43.635084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:43.635087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:43.635360 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:43.635523 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:43.636245 (+ 722us) server_negotiation.cc:299] Negotiation successful
0924 03:09:43.636521 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":342,"threads_started":1}
I20260924 03:09:43.637953 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:43.619389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.3:38743)
0924 03:09:43.620223 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:43.620267 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:43.622275 (+ 2008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:43.623026 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:43.623034 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:43.623721 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:43.624336 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:43.624353 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:43.625484 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:43.625489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:43.626758 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:43.626770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:43.627584 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:43.627592 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:43.627805 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:43.628561 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:43.628593 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:43.630838 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:43.633922 (+ 3084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:43.633931 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:43.633948 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:43.634365 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:43.634762 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:43.634770 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:43.634774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:43.634961 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:43.635491 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:43.635499 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:43.635859 (+ 360us) client_negotiation.cc:769] Sending connection context
0924 03:09:43.636133 (+ 274us) client_negotiation.cc:240] Negotiation successful
0924 03:09:43.636754 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":344,"threads_started":1}
I20260924 03:09:43.639358 7020 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34603
I20260924 03:09:43.639685 7020 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:43.640447 7020 heartbeater.cc:511] Master 127.0.97.62:34603 requested a full tablet report, sending...
I20260924 03:09:43.642376 6563 ts_manager.cc:194] Registered new tserver with Master: 8a3d53cd8e5c47cbb4da460f4198db40 (127.0.97.3:37999)
I20260924 03:09:43.643263 6563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38743
I20260924 03:09:43.649396 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260924 03:09:43.701620 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34973
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34603,127.0.97.61:34973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:44.045095 7026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:44.045397 7026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:44.045501 7026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:44.053387 7026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:44.053505 7026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:44.053556 7026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:44.053611 7026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:44.053658 7026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:09:44.063934 7026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34603,127.0.97.61:34973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
W20260924 03:09:44.064301 7026 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:34603,127.0.97.61:34973'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 03:09:44.065598 7026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:44.066890 7026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:44.079145 7032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:44.079202 7034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:44.079914 7031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:44.079939 7026 server_base.cc:1034] running on GCE node
I20260924 03:09:44.080894 7026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:44.082499 7026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:44.083787 7026 hybrid_clock.cc:648] HybridClock initialized: now 1790219384083703 us; error 61 us; skew 500 ppm
I20260924 03:09:44.084192 7026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:44 dist-test-slave-5l98 krb5kdc[6518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219384, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:44.088130 7026 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260924 03:09:44.090066 7026 webserver.cc:460] Webserver started at http://127.0.97.61:46857/ using document root <none> and password file <none>
I20260924 03:09:44.090828 7026 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:44.090969 7026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:44.091315 7026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:44.094432 7026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/data/instance:
uuid: "782bd86c1d32482598fdc6ed60cc391b"
format_stamp: "Formatted at 2026-09-24 03:09:44 on dist-test-slave-5l98"
server_key: "1faa84c7049cbe698a5dd0ba169a7eba"
server_key_iv: "6ba283c7e52ee4d985d802ac80976ed3"
server_key_version: "encryptionkey@0"
I20260924 03:09:44.095213 7026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/wal/instance:
uuid: "782bd86c1d32482598fdc6ed60cc391b"
format_stamp: "Formatted at 2026-09-24 03:09:44 on dist-test-slave-5l98"
server_key: "1faa84c7049cbe698a5dd0ba169a7eba"
server_key_iv: "6ba283c7e52ee4d985d802ac80976ed3"
server_key_version: "encryptionkey@0"
I20260924 03:09:44.100384 7026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:09:44.104080 7041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:44.105607 7026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:09:44.105803 7026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/wal
uuid: "782bd86c1d32482598fdc6ed60cc391b"
format_stamp: "Formatted at 2026-09-24 03:09:44 on dist-test-slave-5l98"
server_key: "1faa84c7049cbe698a5dd0ba169a7eba"
server_key_iv: "6ba283c7e52ee4d985d802ac80976ed3"
server_key_version: "encryptionkey@0"
I20260924 03:09:44.106182 7026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:44.126751 6886 heartbeater.cc:503] Master 127.0.97.62:34603 was elected leader, sending a full tablet report...
I20260924 03:09:44.131857 7026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:44.136179 7026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:44.136422 7026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:44.156342 7026 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34973
I20260924 03:09:44.156342 7092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34973 every 8 connection(s)
I20260924 03:09:44.157958 7026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/data/info.pb
I20260924 03:09:44.162185 7093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:44.163498 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 7026
I20260924 03:09:44.171469 7093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:09:44 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 24 03:09:44 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219384, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 03:09:44.188439 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.165662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47114 (local address 127.0.97.61:34973)
0924 03:09:44.166489 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:44.166501 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:44.166550 (+ 49us) server_negotiation.cc:407] Connection header received
0924 03:09:44.168198 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:44.168218 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:44.168685 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:44.169027 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:44.170767 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.172074 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.173914 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.175156 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.175940 (+ 784us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.180191 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:44.180202 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:44.180216 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:44.180265 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:44.183480 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.184342 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.184351 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.184361 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.184571 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.184982 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.184989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.184994 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.185681 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:44.185989 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:44.186480 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 03:09:44.186948 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":355,"threads_started":1}
I20260924 03:09:44.191218 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.174561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55646 (local address 127.0.97.62:34603)
0924 03:09:44.176072 (+ 1511us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:44.176078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:44.176098 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:09:44.176158 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:44.176164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:44.176284 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:44.176412 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:44.177673 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.178679 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.180165 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.180818 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.180942 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.184483 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:44.184514 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:44.184518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:44.184571 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:44.187907 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.189067 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.189073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.189077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.189200 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.189590 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.189593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.189595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.189893 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:44.190161 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:44.190614 (+ 453us) server_negotiation.cc:299] Negotiation successful
0924 03:09:44.190837 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1398,"thread_start_us":1667,"threads_started":1}
I20260924 03:09:44.191269 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.174464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.0.1:55646)
0924 03:09:44.175667 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:44.175729 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:44.175960 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:44.176558 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:44.176567 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:44.177163 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:44.177490 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.177509 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.178814 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.178819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.180026 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.180037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.181173 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.181182 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.181400 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.182030 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:44.182057 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:44.184279 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:44.188116 (+ 3837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.188128 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.188136 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.188893 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.189304 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.189309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.189312 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.189480 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.190134 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:44.190142 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:44.190319 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 03:09:44.190535 (+ 216us) client_negotiation.cc:240] Negotiation successful
0924 03:09:44.190842 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"thread_start_us":640,"threads_started":1}
I20260924 03:09:44.192961 7093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:09:44 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219384, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260924 03:09:44.207640 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.193832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34973 (local address 127.0.0.1:47116)
0924 03:09:44.194135 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:44.194155 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:44.194285 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:44.194659 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:44.194663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:44.195194 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:44.195407 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.195417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.196903 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.196918 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.198006 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.198025 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.198887 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.198892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.199132 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.199743 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:44.199767 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:44.202449 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:44.205548 (+ 3099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.205552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.205554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.205995 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.206310 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.206313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.206316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.206462 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.206947 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:44.206950 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:44.207065 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 03:09:44.207175 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 03:09:44.207380 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":22}
I20260924 03:09:44.207908 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.193796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47116 (local address 127.0.97.61:34973)
0924 03:09:44.194015 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:44.194020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:44.194236 (+ 216us) server_negotiation.cc:407] Connection header received
0924 03:09:44.194409 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:44.194415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:44.194498 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:44.194603 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:44.195584 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.196761 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.198147 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.198734 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.198879 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.202639 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:44.202644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:44.202649 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:44.202681 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:44.205403 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.206106 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.206110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.206112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.206213 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.206547 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.206550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.206552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.206825 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:44.206930 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:44.207254 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 03:09:44.207549 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":71}
I20260924 03:09:44.212888 7093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: Bootstrap starting.
I20260924 03:09:44.216243 7093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:44.217379 7093 log.cc:824] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:44.220161 7093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: No bootstrap required, opened a new log
I20260924 03:09:44.227154 7093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } }
I20260924 03:09:44.227475 7093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:44.227538 7093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782bd86c1d32482598fdc6ed60cc391b, State: Initialized, Role: FOLLOWER
I20260924 03:09:44.228052 7093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } }
I20260924 03:09:44.229013 7100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } } }
I20260924 03:09:44.229354 7100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:09:44.229986 7093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [sys.catalog]: configured and running, proceeding with master startup.
W20260924 03:09:44.239375 7111 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:09:44.239527 7111 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:09:44.241709 7026 master_runner.cc:428] Detected that this master 782bd86c1d32482598fdc6ed60cc391b is joining an existing cluster
I20260924 03:09:44.241830 7026 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:34973
I20260924 03:09:44.262775 7117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.246846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34973 (local address 127.0.0.1:47124)
0924 03:09:44.247527 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:44.247553 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:44.247723 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:44.248276 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:44.248284 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:44.249065 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:44.249506 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.249523 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.251066 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.251073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.252873 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.252890 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.253843 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.253847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.253942 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.254607 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:44.254624 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:44.255385 (+ 761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:44.259642 (+ 4257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.259647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.259651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.260180 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.260669 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.260673 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.260677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.260854 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.261675 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:44.261680 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:44.261810 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 03:09:44.261964 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 03:09:44.262270 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":373,"threads_started":1}
I20260924 03:09:44.264180 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.246779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47124 (local address 127.0.97.61:34973)
0924 03:09:44.247000 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:44.247009 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:44.247629 (+ 620us) server_negotiation.cc:407] Connection header received
0924 03:09:44.247904 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:44.247911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:44.247992 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:44.248063 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:44.249656 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.250840 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.253026 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.253527 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.253598 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.255592 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:44.255598 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:44.255603 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:44.255639 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:44.259488 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.260375 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.260380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.260383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.260545 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.261011 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.261017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.261021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.261468 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:44.261626 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:44.263566 (+ 1940us) server_negotiation.cc:299] Negotiation successful
0924 03:09:44.263787 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260924 03:09:44.271560 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.247849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.0.97.62:34603)
0924 03:09:44.248026 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:44.248037 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:44.257165 (+ 9128us) server_negotiation.cc:407] Connection header received
0924 03:09:44.257377 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:44.257672 (+ 295us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:44.257806 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:44.257889 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:44.259423 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.260562 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.262625 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.263292 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.263398 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.265888 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:44.265894 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:44.265898 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:44.265939 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:44.268747 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.269364 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.269372 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.269377 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.269531 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.269904 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.269911 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.269915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.270189 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:44.270355 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:44.270840 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 03:09:44.271062 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 03:09:44.271636 7118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.248408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.0.1:55656)
0924 03:09:44.257022 (+ 8614us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:44.257046 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:44.257176 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:44.258117 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:44.258123 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:44.258874 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:44.259236 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.259249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.260742 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.260748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.262460 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.262474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.263696 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.263703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.263863 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.264560 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:44.264582 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:44.265684 (+ 1102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:44.268899 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.268906 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.268910 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.269241 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.269648 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.269654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.269657 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.269796 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.270378 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:44.270383 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:44.270580 (+ 197us) client_negotiation.cc:769] Sending connection context
0924 03:09:44.270741 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 03:09:44.271230 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8470,"thread_start_us":8255,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 24 03:09:44 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219381, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260924 03:09:44.293174 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.278441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47134 (local address 127.0.97.61:34973)
0924 03:09:44.278644 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:44.278651 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:44.278886 (+ 235us) server_negotiation.cc:407] Connection header received
0924 03:09:44.279269 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:44.279275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:44.279371 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:44.279452 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:44.281112 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.281913 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.283227 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.283725 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.283880 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.287283 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:44.287288 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:44.287291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:44.287324 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:44.290132 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.290859 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.290868 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.290875 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.291016 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.291406 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.291411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.291414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.291769 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:44.291918 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:44.292535 (+ 617us) server_negotiation.cc:299] Negotiation successful
0924 03:09:44.292783 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:09:44.293217 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.278045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34973 (local address 127.0.0.1:47134)
0924 03:09:44.278739 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:44.278779 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:44.279029 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:44.279731 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:44.279738 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:44.280646 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:44.280943 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.280956 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.282061 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.282069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.283074 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.283087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.283901 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.283907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.284111 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.284938 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:44.284961 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:44.287084 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:44.290301 (+ 3217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.290314 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.290322 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.290731 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.291130 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.291136 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.291140 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.291292 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.292002 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:44.292008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:44.292198 (+ 190us) client_negotiation.cc:769] Sending connection context
0924 03:09:44.292392 (+ 194us) client_negotiation.cc:240] Negotiation successful
0924 03:09:44.292775 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":599,"threads_started":1}
I20260924 03:09:44.295408 6563 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:34973: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 34973
}
attrs {
promote: true
}
}
dest_uuid: "0a226d7c36f749c78880160588ba0ba1"
cas_config_opid_index: -1
I20260924 03:09:44.296344 6563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: true } }
I20260924 03:09:44.299451 7124 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 782bd86c1d32482598fdc6ed60cc391b (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: true } } }
I20260924 03:09:44.300671 7068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:44.303939 7124 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:34973
I20260924 03:09:44.304780 7026 master.cc:537] Master@127.0.97.61:34973 shutting down...
I20260924 03:09:44.304594 7123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0a226d7c36f749c78880160588ba0ba1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: true } } }
I20260924 03:09:44.305683 7123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:44.306250 7068 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 1 FOLLOWER]: Refusing update from remote peer 0a226d7c36f749c78880160588ba0ba1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 03:09:44.307267 7123 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260924 03:09:44.309623 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:32857
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34603,127.0.97.61:34973,127.0.97.60:32857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:44.313396 6548 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 03:09:44.318207 6548 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 -> Peer 782bd86c1d32482598fdc6ed60cc391b (127.0.97.61:34973): Couldn't send request to peer 782bd86c1d32482598fdc6ed60cc391b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 03:09:44.318609 7026 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:44.319103 7026 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:44.319281 7026 tablet_replica.cc:330] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: stopping tablet replica
I20260924 03:09:44.325670 7026 master.cc:552] Master@127.0.97.61:34973 shutdown complete.
I20260924 03:09:44.325788 7026 master_runner.cc:305] Clearing existing system tablet
I20260924 03:09:44.326175 7026 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:44.326287 7026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:44.328470 7026 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 03:09:44.330072 7129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:44.330367 7026 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:44.330649 7026 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:09:44.330734 7026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/wal
uuid: "782bd86c1d32482598fdc6ed60cc391b"
format_stamp: "Formatted at 2026-09-24 03:09:44 on dist-test-slave-5l98"
server_key: "1faa84c7049cbe698a5dd0ba169a7eba"
server_key_iv: "6ba283c7e52ee4d985d802ac80976ed3"
server_key_version: "encryptionkey@0"
I20260924 03:09:44.334775 7026 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:44.339692 7026 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 03:09:44.339901 7026 log.cc:1196] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 03:09:44.340282 7026 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: Deleting consensus metadata
I20260924 03:09:44.340585 7026 master_runner.cc:315] Copying system tablet from 127.0.97.62:34603
I20260924 03:09:44.343662 7026 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:34603
I20260924 03:09:44.356959 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.344555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.0.97.62:34603)
0924 03:09:44.344776 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:44.344786 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:44.345296 (+ 510us) server_negotiation.cc:407] Connection header received
0924 03:09:44.345519 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:44.345525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:44.345615 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:44.345733 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:44.346557 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.347519 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.349055 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.349454 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.349600 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.351428 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:44.351434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:44.351438 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:44.351478 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:44.354202 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.354842 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.354848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.354851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.354975 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.355340 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.355348 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.355351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.355671 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:44.355831 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:44.356257 (+ 426us) server_negotiation.cc:299] Negotiation successful
0924 03:09:44.356472 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:09:44.356992 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.344527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.0.1:55668)
0924 03:09:44.345182 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:44.345205 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:44.345367 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:44.345761 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:44.345765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:44.346174 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:44.346396 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.346406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.347700 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.347705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.348901 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.348913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.349610 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.349616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.349801 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.350497 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:44.350516 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:44.351224 (+ 708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:44.354345 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.354351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.354355 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.354711 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.355091 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.355096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.355099 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.355228 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.355798 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:44.355803 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:44.355993 (+ 190us) client_negotiation.cc:769] Sending connection context
0924 03:09:44.356173 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 03:09:44.356473 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":415,"threads_started":1}
I20260924 03:09:44.358188 6583 tablet_copy_service.cc:140] P 0a226d7c36f749c78880160588ba0ba1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 782bd86c1d32482598fdc6ed60cc391b ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:55668)
I20260924 03:09:44.358439 6583 tablet_copy_service.cc:161] P 0a226d7c36f749c78880160588ba0ba1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 782bd86c1d32482598fdc6ed60cc391b at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:55668: session id = 782bd86c1d32482598fdc6ed60cc391b-00000000000000000000000000000000
I20260924 03:09:44.361744 6583 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 03:09:44.363729 7026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 03:09:44.368736 7026 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: tablet copy: Starting download of 0 data blocks...
I20260924 03:09:44.369092 7026 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: tablet copy: Starting download of 1 WAL segments...
I20260924 03:09:44.371918 7026 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 03:09:44.375700 6583 tablet_copy_service.cc:342] P 0a226d7c36f749c78880160588ba0ba1: Request end of tablet copy session 782bd86c1d32482598fdc6ed60cc391b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:55668
I20260924 03:09:44.375917 6583 tablet_copy_service.cc:433] P 0a226d7c36f749c78880160588ba0ba1: ending tablet copy session 782bd86c1d32482598fdc6ed60cc391b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 782bd86c1d32482598fdc6ed60cc391b
I20260924 03:09:44.386011 7026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:44.389106 7136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:44.394972 7137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:44.399607 7139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:44.401011 7026 server_base.cc:1034] running on GCE node
I20260924 03:09:44.401340 7026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:44.402104 7026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:44.403427 7026 hybrid_clock.cc:648] HybridClock initialized: now 1790219384403356 us; error 29 us; skew 500 ppm
I20260924 03:09:44.403728 7026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:44 dist-test-slave-5l98 krb5kdc[6518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219384, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:44.405728 7026 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260924 03:09:44.407080 7026 webserver.cc:460] Webserver started at http://127.0.97.61:34587/ using document root <none> and password file <none>
I20260924 03:09:44.407399 7026 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:44.407495 7026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:44.410130 7026 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 03:09:44.412074 7146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:44.412557 7026 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260924 03:09:44.412726 7026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/wal
uuid: "782bd86c1d32482598fdc6ed60cc391b"
format_stamp: "Formatted at 2026-09-24 03:09:44 on dist-test-slave-5l98"
server_key: "1faa84c7049cbe698a5dd0ba169a7eba"
server_key_iv: "6ba283c7e52ee4d985d802ac80976ed3"
server_key_version: "encryptionkey@0"
I20260924 03:09:44.412899 7026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:44.421897 7026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:44.425607 7026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:44.454897 7026 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34973
I20260924 03:09:44.454945 7197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34973 every 8 connection(s)
I20260924 03:09:44.455973 7026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-1/data/info.pb
I20260924 03:09:44.462972 7198 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:09:44.465839 7198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: Bootstrap starting.
I20260924 03:09:44.478473 7198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:09:44.478965 7198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: Bootstrap complete.
I20260924 03:09:44.479800 7198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: true } }
I20260924 03:09:44.480024 7198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 782bd86c1d32482598fdc6ed60cc391b, State: Initialized, Role: LEARNER
I20260924 03:09:44.480243 7198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: true } }
I20260924 03:09:44.481047 7201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: true } } }
I20260924 03:09:44.481379 7198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:09:44.481426 7201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [sys.catalog]: This master's current role is: LEARNER
I20260924 03:09:44.489619 7212 catalog_manager.cc:1259] Loaded cluster ID: adc5298acf804509a20dfe0c10861d58
I20260924 03:09:44.489760 7212 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: loading cluster ID for follower catalog manager: success
I20260924 03:09:44.492102 7212 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: acquiring CA information for follower catalog manager: success
I20260924 03:09:44.494004 7212 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:09:44.646108 7020 heartbeater.cc:503] Master 127.0.97.62:34603 was elected leader, sending a full tablet report...
W20260924 03:09:44.739372 7127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:44.739792 7127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:44.740001 7127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 03:09:44.748600 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.739578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34973 (local address 127.0.0.1:47148)
0924 03:09:44.739866 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:44.739886 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:44.740015 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:44.741086 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:44.741092 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:44.741123 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:44.741314 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.741323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.743692 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.743698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.745738 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.745754 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.747650 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.747657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.747881 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.747956 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 03:09:44.748056 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 03:09:44.748261 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260924 03:09:44.748641 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.739786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47148 (local address 127.0.97.61:34973)
0924 03:09:44.740694 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:44.740700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:44.740725 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:09:44.740794 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:44.740799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:44.740884 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:44.741006 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:44.741484 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.743496 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.745918 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.747471 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.747604 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.747844 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:44.748107 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 03:09:44.748249 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":586,"threads_started":1}
I20260924 03:09:44.750224 7173 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260924 03:09:44.750833 7201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a226d7c36f749c78880160588ba0ba1. Latest consensus state: current_term: 1 leader_uuid: "0a226d7c36f749c78880160588ba0ba1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: true } } }
I20260924 03:09:44.751087 7201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [sys.catalog]: This master's current role is: LEARNER
W20260924 03:09:44.751159 7127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:44.751282 7127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:44.751360 7127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:44.751443 7127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:44.751540 7127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:09:44.762809 7127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34603,127.0.97.61:34973,127.0.97.60:32857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:32857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:44.764299 7127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:44.765671 7127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:44.779032 7219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:44.779269 7218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:44.779016 7221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:44.780561 7127 server_base.cc:1034] running on GCE node
I20260924 03:09:44.781462 7127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:44.783116 7127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:44.784373 7127 hybrid_clock.cc:648] HybridClock initialized: now 1790219384784337 us; error 58 us; skew 500 ppm
I20260924 03:09:44.784865 7127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:44 dist-test-slave-5l98 krb5kdc[6518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219384, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:44.788054 7127 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260924 03:09:44.789788 7127 webserver.cc:460] Webserver started at http://127.0.97.60:39389/ using document root <none> and password file <none>
I20260924 03:09:44.790462 7127 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:44.790561 7127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:44.790805 7127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:44.794319 7127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/data/instance:
uuid: "33b19f8aecbc4da1b1150ad5360242af"
format_stamp: "Formatted at 2026-09-24 03:09:44 on dist-test-slave-5l98"
server_key: "c7bbe1a0170a8e23b740ba1641023f39"
server_key_iv: "43747bae9f8c3904159b01ede6dab478"
server_key_version: "encryptionkey@0"
I20260924 03:09:44.795089 7127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/wal/instance:
uuid: "33b19f8aecbc4da1b1150ad5360242af"
format_stamp: "Formatted at 2026-09-24 03:09:44 on dist-test-slave-5l98"
server_key: "c7bbe1a0170a8e23b740ba1641023f39"
server_key_iv: "43747bae9f8c3904159b01ede6dab478"
server_key_version: "encryptionkey@0"
I20260924 03:09:44.800637 7127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:09:44.804708 7228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:44.806310 7127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:09:44.806494 7127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/wal
uuid: "33b19f8aecbc4da1b1150ad5360242af"
format_stamp: "Formatted at 2026-09-24 03:09:44 on dist-test-slave-5l98"
server_key: "c7bbe1a0170a8e23b740ba1641023f39"
server_key_iv: "43747bae9f8c3904159b01ede6dab478"
server_key_version: "encryptionkey@0"
I20260924 03:09:44.806681 7127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:44.823342 7127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:44.828150 7127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:44.828456 7127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:44.850687 7127 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:32857
I20260924 03:09:44.850718 7279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:32857 every 8 connection(s)
I20260924 03:09:44.852085 7127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/data/info.pb
I20260924 03:09:44.855918 7280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:44.859716 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 7127
I20260924 03:09:44.867357 7280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:09:44 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219384, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 24 03:09:44 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260924 03:09:44.892298 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.872958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55674 (local address 127.0.97.62:34603)
0924 03:09:44.874273 (+ 1315us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:44.874279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:44.874303 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:09:44.875450 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:44.875455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:44.875536 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:44.875659 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:44.877542 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.878393 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.879956 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.880360 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.880502 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.884593 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:44.884600 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:44.884604 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:44.884784 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:44.888457 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.889900 (+ 1443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.889908 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.889912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.890053 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.890496 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.890501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.890505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.890785 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:44.890948 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:44.891667 (+ 719us) server_negotiation.cc:299] Negotiation successful
0924 03:09:44.891900 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1170,"thread_start_us":1054,"threads_started":1}
I20260924 03:09:44.893373 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.862374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56480 (local address 127.0.97.60:32857)
0924 03:09:44.872712 (+ 10338us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:44.872728 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:44.872773 (+ 45us) server_negotiation.cc:407] Connection header received
0924 03:09:44.875094 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:44.875120 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:44.875378 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:44.875625 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:44.877694 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.878849 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.880104 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.880730 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.880871 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.885707 (+ 4836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:44.885719 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:44.885737 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:44.885776 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:44.889820 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.890525 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.890534 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.890539 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.890674 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.891065 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.891070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.891073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.891425 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:44.891718 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:44.892052 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 03:09:44.892295 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10044,"spinlock_wait_cycles":19712,"thread_start_us":9812,"threads_started":1}
I20260924 03:09:44.893394 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.872324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.0.1:55674)
0924 03:09:44.873409 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:44.873467 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:44.875127 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:44.875723 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:44.875730 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:44.876580 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:44.877249 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.877291 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.878545 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.878551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.879794 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.879805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.880635 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.880643 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.880822 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.881600 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:44.881622 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:44.884295 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:44.888632 (+ 4337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.888640 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.888650 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.889716 (+ 1066us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.890178 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.890184 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.890187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.890371 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.890937 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:44.890945 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:44.891298 (+ 353us) client_negotiation.cc:769] Sending connection context
0924 03:09:44.891581 (+ 283us) client_negotiation.cc:240] Negotiation successful
0924 03:09:44.892097 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":759,"threads_started":1}
I20260924 03:09:44.895051 7280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:09:44 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219384, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260924 03:09:44.908612 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.895742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34973 (local address 127.0.0.1:47150)
0924 03:09:44.895921 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:44.895944 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:44.896075 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:44.896400 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:44.896405 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:44.896928 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:44.897059 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.897067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.898039 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.898047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.898961 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.898971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.899692 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.899698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.899890 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.900551 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:44.900574 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:44.902657 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:44.906109 (+ 3452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.906115 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.906119 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.906566 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.906956 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.906963 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.906966 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.907103 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.907622 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:44.907627 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:44.907774 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 03:09:44.907915 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 03:09:44.908228 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 03:09:44.908635 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.895932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47150 (local address 127.0.97.61:34973)
0924 03:09:44.896130 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:44.896137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:44.896156 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:09:44.896210 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:44.896221 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:44.896278 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:44.896382 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:44.897202 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.897900 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.899106 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.899535 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.899675 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.902858 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:44.902865 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:44.902874 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:44.902907 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:44.905949 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.906700 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.906706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.906709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.906835 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.907210 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.907217 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.907220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.907504 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:44.907657 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:44.908017 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 03:09:44.908258 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:09:44.910280 7280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:09:44 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219384, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260924 03:09:44.923894 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.910934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32857 (local address 127.0.0.1:56490)
0924 03:09:44.911240 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:44.911295 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:44.911402 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:44.911785 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:44.911789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:44.912172 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:44.912304 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.912322 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.913449 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.913457 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.914350 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.914360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.915185 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.915189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.915347 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.915987 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:44.916008 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:44.918572 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:44.921649 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.921653 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.921656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.922035 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.922404 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.922408 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.922410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.922568 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.923107 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:44.923110 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:44.923233 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 03:09:44.923344 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 03:09:44.923566 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260924 03:09:44.923974 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.911218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56490 (local address 127.0.97.60:32857)
0924 03:09:44.911366 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:44.911370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:44.911390 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:09:44.911588 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:44.911592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:44.911646 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:44.911774 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:44.912452 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.913250 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.914541 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.915048 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.915151 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.918771 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:44.918776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:44.918779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:44.918808 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:44.921505 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.922147 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.922151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.922153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.922268 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.922674 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.922677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.922679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.922944 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:44.923052 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:44.923445 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 03:09:44.923683 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":55}
I20260924 03:09:44.928906 7280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af: Bootstrap starting.
I20260924 03:09:44.931911 7280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:44.932935 7280 log.cc:824] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:44.935560 7280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af: No bootstrap required, opened a new log
I20260924 03:09:44.943089 7280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } }
I20260924 03:09:44.943397 7280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:44.943470 7280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33b19f8aecbc4da1b1150ad5360242af, State: Initialized, Role: FOLLOWER
I20260924 03:09:44.943977 7280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } }
I20260924 03:09:44.944882 7287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } } }
I20260924 03:09:44.945230 7287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:09:44.945883 7280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [sys.catalog]: configured and running, proceeding with master startup.
W20260924 03:09:44.955670 7298 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:09:44.955817 7298 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:09:44.960601 7127 master_runner.cc:428] Detected that this master 33b19f8aecbc4da1b1150ad5360242af is joining an existing cluster
I20260924 03:09:44.960736 7127 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:32857
I20260924 03:09:44.986828 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.965540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55682 (local address 127.0.97.62:34603)
0924 03:09:44.965873 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:44.965879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:44.966417 (+ 538us) server_negotiation.cc:407] Connection header received
0924 03:09:44.967632 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:44.967638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:44.967737 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:44.967808 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:44.970582 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.971805 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.974947 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.975623 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.975721 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.979124 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:44.979131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:44.979136 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:44.979193 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:44.983394 (+ 4201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.984401 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.984408 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.984412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.984594 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.985064 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.985069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.985073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.985441 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:44.985565 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:44.986231 (+ 666us) server_negotiation.cc:299] Negotiation successful
0924 03:09:44.986468 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221}
I20260924 03:09:44.987561 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.966979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56494 (local address 127.0.97.60:32857)
0924 03:09:44.967137 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:44.967144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:44.967472 (+ 328us) server_negotiation.cc:407] Connection header received
0924 03:09:44.967711 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:44.967716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:44.967782 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:44.967850 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:44.970644 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.971912 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.975123 (+ 3211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.975951 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.976184 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.979551 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:44.979558 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:44.979562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:44.979597 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:44.983951 (+ 4354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.984929 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.984934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.984938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.985104 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.985605 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.985612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.985615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.986066 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:44.986194 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:44.986794 (+ 600us) server_negotiation.cc:299] Negotiation successful
0924 03:09:44.987064 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":65,"spinlock_wait_cycles":1280}
I20260924 03:09:44.988369 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.965347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.0.97.61:34973)
0924 03:09:44.965736 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:44.965742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:44.967295 (+ 1553us) server_negotiation.cc:407] Connection header received
0924 03:09:44.967481 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:44.967490 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:44.967612 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:44.967717 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:44.970674 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.971908 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.974464 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.975219 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.975327 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.979132 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:44.979141 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:44.979150 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:44.979200 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:44.983788 (+ 4588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.984935 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.984941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.984944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.985104 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:44.985631 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:44.985636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:44.985639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:44.986037 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:44.986179 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:44.986818 (+ 639us) server_negotiation.cc:299] Negotiation successful
0924 03:09:44.987955 (+ 1137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260924 03:09:44.988595 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.965382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.0.1:55682)
0924 03:09:44.966310 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:44.966333 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:44.967493 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:44.968029 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:44.968034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:44.969726 (+ 1692us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:44.970376 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.970396 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.973024 (+ 2628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.973029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.974754 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.974768 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.976024 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.976031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.976299 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.977123 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:44.977145 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:44.978867 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:44.983594 (+ 4727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.983608 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.983612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.984252 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.984745 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.984750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.984754 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.984936 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.985665 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:44.985673 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:44.985922 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 03:09:44.987112 (+ 1190us) client_negotiation.cc:240] Negotiation successful
0924 03:09:44.987469 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":102,"thread_start_us":635,"threads_started":1}
I20260924 03:09:44.989162 7306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.966647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32857 (local address 127.0.0.1:56494)
0924 03:09:44.967344 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:44.967363 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:44.967555 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:44.968086 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:44.968093 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:44.969689 (+ 1596us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:44.970476 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.970490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.972422 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.972427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.974946 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.974961 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.976568 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.976579 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.976795 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.978025 (+ 1230us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:44.978058 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:44.979359 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:44.984101 (+ 4742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.984106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.984110 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.984783 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.985254 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.985260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.985264 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.985483 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.986266 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:44.986270 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:44.986481 (+ 211us) client_negotiation.cc:769] Sending connection context
0924 03:09:44.987827 (+ 1346us) client_negotiation.cc:240] Negotiation successful
0924 03:09:44.988591 (+ 764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":78,"spinlock_wait_cycles":22144,"thread_start_us":413,"threads_started":1}
I20260924 03:09:44.990604 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.965463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34973 (local address 127.0.0.1:47154)
0924 03:09:44.966847 (+ 1384us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:44.967174 (+ 327us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:44.967331 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:44.967912 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:44.967917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:44.969680 (+ 1763us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:44.970476 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.970490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.972145 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.972153 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.974258 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.974274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.975663 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.975669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.975881 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:44.976739 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:44.976765 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:44.978890 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:44.983970 (+ 5080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.983975 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.983986 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.984783 (+ 797us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.985254 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:44.985260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:44.985264 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:44.985483 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:44.986261 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:44.986265 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:44.986519 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 03:09:44.989774 (+ 3255us) client_negotiation.cc:240] Negotiation successful
0924 03:09:44.990113 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1262,"mutex_wait_us":62,"spinlock_wait_cycles":50816,"thread_start_us":1093,"threads_started":1}
I20260924 03:09:45.000653 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 6624
Sep 24 03:09:44 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219381, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260924 03:09:45.011727 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.994801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32857 (local address 127.0.0.1:56498)
0924 03:09:44.995021 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:44.995040 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:44.995150 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:44.995515 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:44.995519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:44.995889 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:44.996146 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.996158 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.997725 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.997729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.998781 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:44.998797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.999717 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.999723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:44.999885 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:45.000476 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:45.000531 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:45.003917 (+ 3386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:45.008452 (+ 4535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:45.008459 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:45.008464 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:45.009084 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:45.009597 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:45.009604 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:45.009608 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:45.009777 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:45.010624 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:45.010629 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:45.010803 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 03:09:45.010965 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 03:09:45.011369 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 03:09:45.012909 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:44.994976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.0.97.60:32857)
0924 03:09:44.995158 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:44.995164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:44.995183 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:09:44.995271 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:44.995276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:44.995366 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:44.995485 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:44.996306 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.997491 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.999016 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:44.999554 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:44.999705 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:45.004257 (+ 4552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:45.004296 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:45.004301 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:45.004353 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:45.008236 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:45.009268 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:45.009273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:45.009278 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:45.009443 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:45.010003 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:45.010008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:45.010012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:45.010467 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:45.012053 (+ 1586us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:45.012213 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 03:09:45.012543 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:09:45.014735 6563 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:32857: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 32857
}
attrs {
promote: true
}
}
dest_uuid: "0a226d7c36f749c78880160588ba0ba1"
cas_config_opid_index: 5
I20260924 03:09:45.015770 6563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: true } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: true } }
Sep 24 03:09:45 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260924 03:09:45.017777 7308 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 33b19f8aecbc4da1b1150ad5360242af (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: true } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: true } } }
I20260924 03:09:45.018486 7173 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 1 LEARNER]: Refusing update from remote peer 0a226d7c36f749c78880160588ba0ba1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 03:09:45.019172 7123 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 03:09:45.021636 7308 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:32857
I20260924 03:09:45.022351 7255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:45.026677 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:45.002481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.0.97.1:37927)
0924 03:09:45.003450 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:45.003460 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:45.003491 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:09:45.003558 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:45.003567 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:45.003870 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:45.004057 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:45.005990 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.007630 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.010001 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.011070 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.011208 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:45.018451 (+ 7243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:45.018469 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:45.018489 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:45.018544 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:45.023079 (+ 4535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:45.024004 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:45.024014 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:45.024018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:45.024190 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:45.024647 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:45.024653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:45.024657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:45.025135 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:45.025421 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:45.025944 (+ 523us) server_negotiation.cc:299] Negotiation successful
0924 03:09:45.026237 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":469,"threads_started":1}
I20260924 03:09:45.028144 6747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:45.028110 7123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0a226d7c36f749c78880160588ba0ba1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: true } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: true } } }
I20260924 03:09:45.028431 7123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:45.034286 7173 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 33b19f8aecbc4da1b1150ad5360242af (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: true } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: true } } }
I20260924 03:09:45.039179 7201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a226d7c36f749c78880160588ba0ba1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: true } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: true } } }
I20260924 03:09:45.039489 7201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [sys.catalog]: This master's current role is: LEARNER
I20260924 03:09:45.039969 7309 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1: attempting to promote NON_VOTER 782bd86c1d32482598fdc6ed60cc391b to VOTER
I20260924 03:09:45.040774 7309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: true } }
I20260924 03:09:45.043496 7173 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 1 LEARNER]: Refusing update from remote peer 0a226d7c36f749c78880160588ba0ba1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 03:09:45.044090 7123 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 03:09:45.046582 7309 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 782bd86c1d32482598fdc6ed60cc391b (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: true } } }
I20260924 03:09:45.047250 7173 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 782bd86c1d32482598fdc6ed60cc391b (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: true } } }
I20260924 03:09:45.048908 7127 master.cc:537] Master@127.0.97.60:32857 shutting down...
I20260924 03:09:45.048990 7255 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [term 1 FOLLOWER]: Refusing update from remote peer 0a226d7c36f749c78880160588ba0ba1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 03:09:45.051120 7254 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [term 1 FOLLOWER]: Refusing update from remote peer 0a226d7c36f749c78880160588ba0ba1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 03:09:45.052549 7308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0a226d7c36f749c78880160588ba0ba1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: true } } }
I20260924 03:09:45.052821 7308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:45.054777 7308 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:45.062424 7309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a226d7c36f749c78880160588ba0ba1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: true } } }
I20260924 03:09:45.062769 7309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:45.064733 7201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a226d7c36f749c78880160588ba0ba1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: true } } }
I20260924 03:09:45.065107 7201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [sys.catalog]: This master's current role is: FOLLOWER
W20260924 03:09:45.067138 6547 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 -> Peer 33b19f8aecbc4da1b1150ad5360242af (127.0.97.60:32857): Couldn't send request to peer 33b19f8aecbc4da1b1150ad5360242af. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 03:09:45.068701 7127 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:45.069150 7127 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:45.069355 7127 tablet_replica.cc:330] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af: stopping tablet replica
I20260924 03:09:45.086125 7127 master.cc:552] Master@127.0.97.60:32857 shutdown complete.
I20260924 03:09:45.086229 7127 master_runner.cc:305] Clearing existing system tablet
I20260924 03:09:45.086650 7127 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:45.086763 7127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:45.090237 7127 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 03:09:45.093690 7315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:45.094111 7127 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:45.094455 7127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:09:45.094568 7127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/wal
uuid: "33b19f8aecbc4da1b1150ad5360242af"
format_stamp: "Formatted at 2026-09-24 03:09:44 on dist-test-slave-5l98"
server_key: "c7bbe1a0170a8e23b740ba1641023f39"
server_key_iv: "43747bae9f8c3904159b01ede6dab478"
server_key_version: "encryptionkey@0"
I20260924 03:09:45.099587 7127 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:45.107110 7127 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 03:09:45.107296 7127 log.cc:1196] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 03:09:45.107755 7127 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af: Deleting consensus metadata
I20260924 03:09:45.108016 7127 master_runner.cc:315] Copying system tablet from 127.0.97.62:34603
I20260924 03:09:45.117017 7127 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:34603
I20260924 03:09:45.132838 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:45.117904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55684 (local address 127.0.97.62:34603)
0924 03:09:45.118122 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:45.118128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:45.118655 (+ 527us) server_negotiation.cc:407] Connection header received
0924 03:09:45.118864 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:45.118870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:45.118965 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:45.119071 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:45.120114 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.121217 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.123056 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.123699 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.123823 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:45.126232 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:45.126238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:45.126244 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:45.126284 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:45.129861 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:45.130642 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:45.130648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:45.130652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:45.130791 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:45.131231 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:45.131236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:45.131240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:45.131602 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:45.131754 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:45.132188 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 03:09:45.132418 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 03:09:45.132870 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:45.117880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.0.1:55684)
0924 03:09:45.118520 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:45.118545 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:45.118705 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:45.119127 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:45.119133 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:45.119667 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:45.119945 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:45.119957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.121359 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.121365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:45.122889 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:45.122903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.123856 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.123862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:45.124055 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:45.124822 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:45.124846 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:45.126006 (+ 1160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:45.130027 (+ 4021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:45.130032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:45.130037 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:45.130510 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:45.130931 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:45.130936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:45.130940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:45.131105 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:45.131743 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:45.131748 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:45.131917 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 03:09:45.132071 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 03:09:45.132424 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":370,"threads_started":1}
I20260924 03:09:45.133818 6583 tablet_copy_service.cc:140] P 0a226d7c36f749c78880160588ba0ba1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 33b19f8aecbc4da1b1150ad5360242af ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:55684)
I20260924 03:09:45.133980 6583 tablet_copy_service.cc:161] P 0a226d7c36f749c78880160588ba0ba1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 33b19f8aecbc4da1b1150ad5360242af at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:55684: session id = 33b19f8aecbc4da1b1150ad5360242af-00000000000000000000000000000000
I20260924 03:09:45.137532 6583 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 03:09:45.139425 7127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 03:09:45.144845 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 6624
I20260924 03:09:45.150384 7127 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af: tablet copy: Starting download of 0 data blocks...
I20260924 03:09:45.150692 7127 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af: tablet copy: Starting download of 1 WAL segments...
I20260924 03:09:45.152966 7127 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 03:09:45.156169 6583 tablet_copy_service.cc:342] P 0a226d7c36f749c78880160588ba0ba1: Request end of tablet copy session 33b19f8aecbc4da1b1150ad5360242af-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:55684
I20260924 03:09:45.156316 6583 tablet_copy_service.cc:433] P 0a226d7c36f749c78880160588ba0ba1: ending tablet copy session 33b19f8aecbc4da1b1150ad5360242af-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 33b19f8aecbc4da1b1150ad5360242af
I20260924 03:09:45.167721 7127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:09:45.168475 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:37927
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34603
--webserver_port=45307
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:34603,127.0.97.61:34973
--tserver_master_addrs=127.0.97.62:34603,127.0.97.61:34973,127.0.97.60:32857 with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:45.174701 7322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:45.175365 7323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:45.181143 7127 server_base.cc:1034] running on GCE node
W20260924 03:09:45.181504 7325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:45.182013 7127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:45.182673 7127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:45.183863 7127 hybrid_clock.cc:648] HybridClock initialized: now 1790219385183819 us; error 32 us; skew 500 ppm
I20260924 03:09:45.184113 7127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:45 dist-test-slave-5l98 krb5kdc[6518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219385, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:45.187269 7127 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260924 03:09:45.201831 7127 webserver.cc:460] Webserver started at http://127.0.97.60:46709/ using document root <none> and password file <none>
I20260924 03:09:45.202158 7127 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:45.202253 7127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:45.205744 7127 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 03:09:45.207700 7333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:45.208192 7127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:09:45.208357 7127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/wal
uuid: "33b19f8aecbc4da1b1150ad5360242af"
format_stamp: "Formatted at 2026-09-24 03:09:44 on dist-test-slave-5l98"
server_key: "c7bbe1a0170a8e23b740ba1641023f39"
server_key_iv: "43747bae9f8c3904159b01ede6dab478"
server_key_version: "encryptionkey@0"
I20260924 03:09:45.208662 7127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:45.230705 7127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:45.235504 7127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:45.259517 7127 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:32857
I20260924 03:09:45.259568 7384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:32857 every 8 connection(s)
I20260924 03:09:45.260844 7127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/master-2/data/info.pb
I20260924 03:09:45.273566 7385 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:09:45.278139 7385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af: Bootstrap starting.
I20260924 03:09:45.299567 7385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:09:45.300318 7385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af: Bootstrap complete.
I20260924 03:09:45.301522 7385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: true } }
I20260924 03:09:45.301822 7385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 33b19f8aecbc4da1b1150ad5360242af, State: Initialized, Role: LEARNER
I20260924 03:09:45.302114 7385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: true } }
I20260924 03:09:45.303325 7385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:09:45.303035 7388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: true } } }
I20260924 03:09:45.303629 7388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [sys.catalog]: This master's current role is: LEARNER
I20260924 03:09:45.311883 7399 catalog_manager.cc:1259] Loaded cluster ID: adc5298acf804509a20dfe0c10861d58
I20260924 03:09:45.311968 7399 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af: loading cluster ID for follower catalog manager: success
I20260924 03:09:45.315074 7399 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af: acquiring CA information for follower catalog manager: success
I20260924 03:09:45.317158 7399 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:09:45.483816 7400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:45.470831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56506 (local address 127.0.97.60:32857)
0924 03:09:45.475640 (+ 4809us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:45.475647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:45.475678 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:09:45.475760 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:45.475766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:45.475865 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:45.476092 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:45.476666 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.478305 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.481075 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.482540 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.482693 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:45.482951 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:45.483250 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 03:09:45.483395 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4644,"thread_start_us":4423,"threads_started":1}
I20260924 03:09:45.483811 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:45.469975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32857 (local address 127.0.0.1:56506)
0924 03:09:45.470355 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:45.470377 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:45.470516 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:45.476170 (+ 5654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:45.476177 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:45.476222 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:45.476434 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:45.476457 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.478511 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.478518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:45.480848 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:45.480870 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.482759 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.482767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:45.482993 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:45.483045 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 03:09:45.483183 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 03:09:45.483383 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241}
I20260924 03:09:45.485498 7360 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260924 03:09:45.486178 7388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a226d7c36f749c78880160588ba0ba1. Latest consensus state: current_term: 1 leader_uuid: "0a226d7c36f749c78880160588ba0ba1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: true } } }
I20260924 03:09:45.486475 7388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [sys.catalog]: This master's current role is: LEARNER
W20260924 03:09:45.631372 7327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:45.631691 7327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:45.631788 7327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:45.640007 7327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:45.640134 7327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:45.640249 7327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:09:45.650875 7327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:37927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45307
--tserver_master_addrs=127.0.97.62:34603,127.0.97.61:34973,127.0.97.60:32857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:45.652297 7327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:45.653760 7327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:45.668689 7405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:45.668713 7406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:45.669123 7408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:45.669485 7327 server_base.cc:1034] running on GCE node
I20260924 03:09:45.670271 7327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:45.671278 7327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:45.672580 7327 hybrid_clock.cc:648] HybridClock initialized: now 1790219385672520 us; error 47 us; skew 500 ppm
I20260924 03:09:45.672907 7327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:45 dist-test-slave-5l98 krb5kdc[6518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219385, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:45.676931 7327 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260924 03:09:45.679256 7327 webserver.cc:460] Webserver started at http://127.0.97.1:45307/ using document root <none> and password file <none>
I20260924 03:09:45.680078 7327 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:45.680204 7327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:45.686494 7327 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 03:09:45.689642 7415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:45.691313 7327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:09:45.691522 7327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "00cb9dff7a9b430c9ad5224aa776edee"
format_stamp: "Formatted at 2026-09-24 03:09:42 on dist-test-slave-5l98"
server_key: "ee60c7f56658d106704940694ece96b9"
server_key_iv: "c731830e7a501ca7819487702e49d475"
server_key_version: "encryptionkey@0"
I20260924 03:09:45.692634 7327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:45.722337 7327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:45.727352 7327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:45.728080 7327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:45.729192 7327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:45.730691 7327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:45.730839 7327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:45.730983 7327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:45.731079 7327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:45.770503 7327 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37927
I20260924 03:09:45.770601 7527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37927 every 8 connection(s)
I20260924 03:09:45.772106 7327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:09:45.772660 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 7327
I20260924 03:09:45.773026 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 6758
Sep 24 03:09:45 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Sep 24 03:09:45 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219385, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 24 03:09:45 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219385, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 24 03:09:45 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219385, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260924 03:09:45.807078 7532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:45.774498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41202 (local address 127.0.97.2:43547)
0924 03:09:45.780399 (+ 5901us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:45.780411 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:45.781734 (+ 1323us) server_negotiation.cc:407] Connection header received
0924 03:09:45.781827 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:45.781877 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:45.782160 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:45.782356 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:45.784828 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.786598 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.788663 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.789629 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.789755 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:45.798931 (+ 9176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:45.798951 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:45.798984 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:45.799036 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:45.803409 (+ 4373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:45.804277 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:45.804285 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:45.804289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:45.804426 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:45.804967 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:45.804974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:45.804979 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:45.805390 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:45.805667 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:45.806310 (+ 643us) server_negotiation.cc:299] Negotiation successful
0924 03:09:45.806644 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5731,"thread_start_us":5589,"threads_started":1}
I20260924 03:09:45.807904 7400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:45.781327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46459 (local address 127.0.97.60:32857)
0924 03:09:45.781545 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:45.781552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:45.783266 (+ 1714us) server_negotiation.cc:407] Connection header received
0924 03:09:45.783762 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:45.783768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:45.783859 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:45.783955 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:45.787343 (+ 3388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.788409 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.791667 (+ 3258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.792338 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.792454 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:45.799188 (+ 6734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:45.799195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:45.799200 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:45.799240 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:45.803032 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:45.804771 (+ 1739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:45.804778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:45.804782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:45.804910 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:45.805670 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:45.805675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:45.805678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:45.806099 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:45.806236 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:45.807349 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0924 03:09:45.807605 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 03:09:45.808202 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:45.776259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34973 (local address 127.0.97.1:37921)
0924 03:09:45.782262 (+ 6003us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:45.782283 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:45.783424 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:45.784455 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:45.784468 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:45.785634 (+ 1166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:45.786637 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:45.786668 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.788229 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.788235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:45.791507 (+ 3272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:45.791528 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.792939 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.792950 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:45.793277 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:45.794388 (+ 1111us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:45.794441 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:45.798593 (+ 4152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:45.803670 (+ 5077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:45.803675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:45.803679 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:45.804189 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:45.804626 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:45.804631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:45.804635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:45.804880 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:45.805601 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:45.805608 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:45.806074 (+ 466us) client_negotiation.cc:769] Sending connection context
0924 03:09:45.806380 (+ 306us) client_negotiation.cc:240] Negotiation successful
0924 03:09:45.806924 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5376,"mutex_wait_us":2425,"spinlock_wait_cycles":74368,"thread_start_us":2592,"threads_started":1}
I20260924 03:09:45.808568 6881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:45.808979 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:45.781522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32857 (local address 127.0.97.1:46459)
0924 03:09:45.783119 (+ 1597us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:45.783142 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:45.783453 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:45.784457 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:45.784470 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:45.785556 (+ 1086us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:45.787172 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:45.787187 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.788623 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.788626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:45.791440 (+ 2814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:45.791455 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.792844 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.792852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:45.793267 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:45.794388 (+ 1121us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:45.794428 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:45.798950 (+ 4522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:45.803257 (+ 4307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:45.803267 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:45.803287 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:45.803980 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:45.805338 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:45.805343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:45.805345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:45.805546 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:45.806894 (+ 1348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:45.806900 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:45.807023 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 03:09:45.808122 (+ 1099us) client_negotiation.cc:240] Negotiation successful
0924 03:09:45.808437 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1448,"mutex_wait_us":428,"spinlock_wait_cycles":58112,"thread_start_us":1218,"threads_started":1}
I20260924 03:09:45.809438 7535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:45.776643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37921 (local address 127.0.97.61:34973)
0924 03:09:45.780039 (+ 3396us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:45.780045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:45.782441 (+ 2396us) server_negotiation.cc:407] Connection header received
0924 03:09:45.783848 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:45.783854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:45.783989 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:45.784085 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:45.786852 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.788035 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.791699 (+ 3664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.792431 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.792592 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:45.798866 (+ 6274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:45.798873 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:45.798877 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:45.798925 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:45.803516 (+ 4591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:45.804340 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:45.804346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:45.804349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:45.804518 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:45.805023 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:45.805029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:45.805035 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:45.805378 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:45.805516 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:45.808781 (+ 3265us) server_negotiation.cc:299] Negotiation successful
0924 03:09:45.809052 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3269,"thread_start_us":3138,"threads_started":1}
I20260924 03:09:45.810501 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:45.777338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56199 (local address 127.0.97.62:34603)
0924 03:09:45.779026 (+ 1688us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:45.779051 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:45.779081 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:09:45.783848 (+ 4767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:45.783854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:45.783940 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:45.784054 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:45.787300 (+ 3246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.788398 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.791696 (+ 3298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.792343 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.792443 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:45.798813 (+ 6370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:45.798823 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:45.798831 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:45.798890 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:45.803097 (+ 4207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:45.804719 (+ 1622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:45.804728 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:45.804735 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:45.804889 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:45.805307 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:45.805314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:45.805320 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:45.805705 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:45.805832 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:45.808979 (+ 3147us) server_negotiation.cc:299] Negotiation successful
0924 03:09:45.809222 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1566,"thread_start_us":1597,"threads_started":1}
I20260924 03:09:45.810665 7534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:45.776259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.1:56199)
0924 03:09:45.777259 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:45.777318 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:45.783424 (+ 6106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:45.784455 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:45.784468 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:45.785556 (+ 1088us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:45.786688 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:45.786703 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.788617 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.788622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:45.791539 (+ 2917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:45.791553 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:45.792852 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:45.792858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:45.793283 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:45.794388 (+ 1105us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:45.794425 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:45.798473 (+ 4048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:45.804098 (+ 5625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:45.804103 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:45.804107 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:45.804615 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:45.805035 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:45.805041 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:45.805045 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:45.805182 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:45.806306 (+ 1124us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:45.806310 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:45.806587 (+ 277us) client_negotiation.cc:769] Sending connection context
0924 03:09:45.806723 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 03:09:45.809757 (+ 3034us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":3618,"spinlock_wait_cycles":61568,"thread_start_us":3448,"threads_started":1}
I20260924 03:09:45.810854 7530 heartbeater.cc:347] Connected to a master server at 127.0.97.61:34973
I20260924 03:09:45.811228 7530 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:45.811707 7528 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34603
I20260924 03:09:45.811852 7528 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:45.812417 7528 heartbeater.cc:511] Master 127.0.97.62:34603 requested a full tablet report, sending...
I20260924 03:09:45.812594 7529 heartbeater.cc:347] Connected to a master server at 127.0.97.60:32857
I20260924 03:09:45.812737 7529 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:45.813305 7529 heartbeater.cc:511] Master 127.0.97.60:32857 requested a full tablet report, sending...
I20260924 03:09:45.812377 7530 heartbeater.cc:511] Master 127.0.97.61:34973 requested a full tablet report, sending...
I20260924 03:09:45.814353 6563 ts_manager.cc:194] Re-registered known tserver with Master: 00cb9dff7a9b430c9ad5224aa776edee (127.0.97.1:37927)
I20260924 03:09:45.815061 7350 ts_manager.cc:194] Registered new tserver with Master: 00cb9dff7a9b430c9ad5224aa776edee (127.0.97.1:37927)
I20260924 03:09:45.815461 6563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56199
I20260924 03:09:45.816391 7163 ts_manager.cc:194] Registered new tserver with Master: 00cb9dff7a9b430c9ad5224aa776edee (127.0.97.1:37927)
I20260924 03:09:45.908382 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 6758
I20260924 03:09:45.925489 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:43547
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34603
--webserver_port=38691
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:34603,127.0.97.61:34973
--tserver_master_addrs=127.0.97.62:34603,127.0.97.61:34973,127.0.97.60:32857 with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:09:46.037431 7309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a226d7c36f749c78880160588ba0ba1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: true } } }
I20260924 03:09:46.037806 7309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:46.039268 7309 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1: attempting to promote NON_VOTER 33b19f8aecbc4da1b1150ad5360242af to VOTER
I20260924 03:09:46.040025 7309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: false } }
I20260924 03:09:46.042193 7173 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 1 FOLLOWER]: Refusing update from remote peer 0a226d7c36f749c78880160588ba0ba1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 03:09:46.042316 7360 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [term 1 LEARNER]: Refusing update from remote peer 0a226d7c36f749c78880160588ba0ba1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 03:09:46.042928 7541 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 03:09:46.043322 7542 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 03:09:46.046599 7541 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 33b19f8aecbc4da1b1150ad5360242af (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: false } } }
I20260924 03:09:46.047308 7173 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 33b19f8aecbc4da1b1150ad5360242af (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: false } } }
I20260924 03:09:46.050521 7545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a226d7c36f749c78880160588ba0ba1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: false } } }
I20260924 03:09:46.050945 7545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:09:46.051692 7542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0a226d7c36f749c78880160588ba0ba1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: false } } }
I20260924 03:09:46.051977 7542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:46.051844 7360 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 33b19f8aecbc4da1b1150ad5360242af (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: false } } }
I20260924 03:09:46.052363 7541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a226d7c36f749c78880160588ba0ba1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: false } } }
I20260924 03:09:46.052428 7542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a226d7c36f749c78880160588ba0ba1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: false } } }
I20260924 03:09:46.052662 7541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:46.052719 7542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:46.055526 7546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a226d7c36f749c78880160588ba0ba1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: false } } }
I20260924 03:09:46.055831 7546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [sys.catalog]: This master's current role is: FOLLOWER
W20260924 03:09:46.283838 7540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:46.284149 7540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:46.284297 7540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:46.291865 7540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:46.291996 7540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:46.292110 7540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:09:46.302088 7540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:43547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=38691
--tserver_master_addrs=127.0.97.62:34603,127.0.97.61:34973,127.0.97.60:32857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:46.303527 7540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:46.304951 7540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:46.320101 7554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:46.320137 7552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:46.320170 7551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:46.321357 7540 server_base.cc:1034] running on GCE node
I20260924 03:09:46.322178 7540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:46.323329 7540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:46.324662 7540 hybrid_clock.cc:648] HybridClock initialized: now 1790219386324591 us; error 62 us; skew 500 ppm
I20260924 03:09:46.325023 7540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:46 dist-test-slave-5l98 krb5kdc[6518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219386, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:46.328562 7540 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260924 03:09:46.330523 7540 webserver.cc:460] Webserver started at http://127.0.97.2:38691/ using document root <none> and password file <none>
I20260924 03:09:46.331384 7540 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:46.331532 7540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:46.337282 7540 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:09:46.340281 7561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:46.341885 7540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:09:46.342092 7540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "765e21fdfb6049d48dcb457debd0e02f"
format_stamp: "Formatted at 2026-09-24 03:09:42 on dist-test-slave-5l98"
server_key: "b1f414bba8013fbc1877d965977533c1"
server_key_iv: "f12b47d6f0f3445c4c047c1311c8d4ea"
server_key_version: "encryptionkey@0"
I20260924 03:09:46.343120 7540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:46.357667 7540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:46.363152 7540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:46.363399 7540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:46.364359 7540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:46.365994 7540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:46.366082 7540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:46.366154 7540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:46.366209 7540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:46.405764 7540 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43547
I20260924 03:09:46.405900 7673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43547 every 8 connection(s)
I20260924 03:09:46.407411 7540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:09:46.409410 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 7540
I20260924 03:09:46.409713 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 6892
Sep 24 03:09:46 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219386, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 24 03:09:46 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219386, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 24 03:09:46 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219386, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 24 03:09:46 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260924 03:09:46.459945 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:46.412266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36825 (local address 127.0.97.60:32857)
0924 03:09:46.431613 (+ 19347us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:46.431622 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:46.431652 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:09:46.434482 (+ 2830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:46.434492 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:46.434614 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:46.434845 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:46.438131 (+ 3286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:46.439803 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:46.443245 (+ 3442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:46.443716 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:46.443795 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:46.451078 (+ 7283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:46.451085 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:46.451091 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:46.451136 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:46.455809 (+ 4673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:46.456918 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:46.456925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:46.456931 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:46.457109 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:46.457690 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:46.457696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:46.457699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:46.458042 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:46.458208 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:46.459277 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0924 03:09:46.459524 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19200,"thread_start_us":19108,"threads_started":1}
I20260924 03:09:46.462344 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:46.411537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32857 (local address 127.0.97.2:36825)
0924 03:09:46.429415 (+ 17878us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:46.429479 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:46.434073 (+ 4594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:46.435514 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:46.435533 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:46.436843 (+ 1310us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:46.437857 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:46.437886 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:46.440074 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:46.440083 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:46.443056 (+ 2973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:46.443068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:46.444024 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:46.444029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:46.444146 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:46.446811 (+ 2665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:46.446857 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:46.450828 (+ 3971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:46.456005 (+ 5177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:46.456018 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:46.456042 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:46.456766 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:46.457282 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:46.457287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:46.457290 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:46.457555 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:46.458308 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:46.458318 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:46.458804 (+ 486us) client_negotiation.cc:769] Sending connection context
0924 03:09:46.460204 (+ 1400us) client_negotiation.cc:240] Negotiation successful
0924 03:09:46.460824 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17396,"mutex_wait_us":38,"spinlock_wait_cycles":31488,"thread_start_us":21022,"threads_started":1}
I20260924 03:09:46.463778 7679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:46.410923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.0.97.3:37999)
0924 03:09:46.432141 (+ 21218us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:46.432150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:46.434080 (+ 1930us) server_negotiation.cc:407] Connection header received
0924 03:09:46.434173 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:46.434182 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:46.434476 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:46.434682 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:46.436862 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:46.438434 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:46.440281 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:46.441206 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:46.441347 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:46.454683 (+ 13336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:46.454702 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:46.454723 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:46.454769 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:46.458647 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:46.459604 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:46.459616 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:46.459620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:46.459772 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:46.460663 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:46.460668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:46.460671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:46.461064 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:46.461352 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:46.463201 (+ 1849us) server_negotiation.cc:299] Negotiation successful
0924 03:09:46.463463 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21043,"thread_start_us":20892,"threads_started":1}
I20260924 03:09:46.464025 7675 heartbeater.cc:347] Connected to a master server at 127.0.97.60:32857
I20260924 03:09:46.464383 7675 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:46.464805 7684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:46.433626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36971 (local address 127.0.97.62:34603)
0924 03:09:46.434233 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:46.434239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:46.441021 (+ 6782us) server_negotiation.cc:407] Connection header received
0924 03:09:46.441211 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:46.441217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:46.441303 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:46.441410 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:46.443241 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:46.444323 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:46.445836 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:46.446269 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:46.446445 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:46.452037 (+ 5592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:46.452043 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:46.452060 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:46.452116 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:46.457957 (+ 5841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:46.459553 (+ 1596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:46.459560 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:46.459565 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:46.459714 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:46.460877 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:46.460883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:46.460887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:46.461247 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:46.461378 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:46.463060 (+ 1682us) server_negotiation.cc:299] Negotiation successful
0924 03:09:46.464231 (+ 1171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":324,"threads_started":1}
I20260924 03:09:46.465358 7675 heartbeater.cc:511] Master 127.0.97.60:32857 requested a full tablet report, sending...
I20260924 03:09:46.465538 7015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:46.465950 7685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:46.434204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.2:36971)
0924 03:09:46.440895 (+ 6691us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:46.440925 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:46.441085 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:46.441446 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:46.441450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:46.442192 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:46.443057 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:46.443068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:46.444589 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:46.444595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:46.445694 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:46.445706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:46.446614 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:46.446625 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:46.446859 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:46.447674 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:46.447704 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:46.451631 (+ 3927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:46.458856 (+ 7225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:46.458861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:46.458865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:46.459411 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:46.459852 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:46.459857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:46.459862 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:46.460036 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:46.461535 (+ 1499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:46.461540 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:46.461689 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 03:09:46.465192 (+ 3503us) client_negotiation.cc:240] Negotiation successful
0924 03:09:46.465531 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6507,"thread_start_us":6978,"threads_started":1}
I20260924 03:09:46.467051 7674 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34603
I20260924 03:09:46.467176 7674 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:46.467585 7350 ts_manager.cc:194] Registered new tserver with Master: 765e21fdfb6049d48dcb457debd0e02f (127.0.97.2:43547)
I20260924 03:09:46.467594 7674 heartbeater.cc:511] Master 127.0.97.62:34603 requested a full tablet report, sending...
I20260924 03:09:46.467664 7680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:46.411339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48393 (local address 127.0.97.61:34973)
0924 03:09:46.431235 (+ 19896us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:46.431244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:46.433308 (+ 2064us) server_negotiation.cc:407] Connection header received
0924 03:09:46.435126 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:46.435132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:46.435220 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:46.435323 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:46.438070 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:46.439161 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:46.441881 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:46.442613 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:46.442693 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:46.450974 (+ 8281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:46.450984 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:46.450989 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:46.451053 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:46.456147 (+ 5094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:46.460174 (+ 4027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:46.460181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:46.460185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:46.460402 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:46.461921 (+ 1519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:46.461927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:46.461931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:46.462313 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:46.462456 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:46.467052 (+ 4596us) server_negotiation.cc:299] Negotiation successful
0924 03:09:46.467305 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19739,"thread_start_us":19580,"threads_started":1}
I20260924 03:09:46.468623 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:46.411776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34973 (local address 127.0.97.2:48393)
0924 03:09:46.433082 (+ 21306us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:46.433104 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:46.434941 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:46.435513 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:46.435531 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:46.436792 (+ 1261us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:46.437854 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:46.437883 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:46.439391 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:46.439396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:46.441693 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:46.441712 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:46.443006 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:46.443011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:46.443276 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:46.446822 (+ 3546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:46.446850 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:46.450642 (+ 3792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:46.458159 (+ 7517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:46.458166 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:46.458173 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:46.458738 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:46.460586 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:46.460591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:46.460596 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:46.460750 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:46.462541 (+ 1791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:46.462546 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:46.462710 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 03:09:46.467831 (+ 5121us) client_negotiation.cc:240] Negotiation successful
0924 03:09:46.468151 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21156,"mutex_wait_us":5,"spinlock_wait_cycles":11776,"thread_start_us":16988,"threads_started":1}
I20260924 03:09:46.468844 6563 ts_manager.cc:194] Re-registered known tserver with Master: 765e21fdfb6049d48dcb457debd0e02f (127.0.97.2:43547)
I20260924 03:09:46.470120 6563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36971
I20260924 03:09:46.471242 7676 heartbeater.cc:347] Connected to a master server at 127.0.97.61:34973
I20260924 03:09:46.471374 7676 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:46.472148 7676 heartbeater.cc:511] Master 127.0.97.61:34973 requested a full tablet report, sending...
I20260924 03:09:46.473297 7163 ts_manager.cc:194] Registered new tserver with Master: 765e21fdfb6049d48dcb457debd0e02f (127.0.97.2:43547)
I20260924 03:09:46.561745 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 6892
I20260924 03:09:46.585642 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37999
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34603
--webserver_port=42127
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:34603,127.0.97.61:34973
--tserver_master_addrs=127.0.97.62:34603,127.0.97.61:34973,127.0.97.60:32857 with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:09:46.819367 7528 heartbeater.cc:503] Master 127.0.97.62:34603 was elected leader, sending a full tablet report...
W20260924 03:09:46.937837 7687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:46.938133 7687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:46.938297 7687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:46.945886 7687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:46.946045 7687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:46.946259 7687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:09:46.957382 7687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=42127
--tserver_master_addrs=127.0.97.62:34603,127.0.97.61:34973,127.0.97.60:32857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:46.958984 7687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:46.960598 7687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:46.975476 7692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:46.975571 7695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:46.975571 7693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:46.976455 7687 server_base.cc:1034] running on GCE node
I20260924 03:09:46.977147 7687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:46.978152 7687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:46.979355 7687 hybrid_clock.cc:648] HybridClock initialized: now 1790219386979289 us; error 64 us; skew 500 ppm
I20260924 03:09:46.979725 7687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:46 dist-test-slave-5l98 krb5kdc[6518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219386, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:46.987787 7687 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260924 03:09:46.989619 7687 webserver.cc:460] Webserver started at http://127.0.97.3:42127/ using document root <none> and password file <none>
I20260924 03:09:46.990339 7687 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:46.990512 7687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:46.996125 7687 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 03:09:46.999076 7702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:47.000629 7687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:09:47.000802 7687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "8a3d53cd8e5c47cbb4da460f4198db40"
format_stamp: "Formatted at 2026-09-24 03:09:43 on dist-test-slave-5l98"
server_key: "4fa4215cb93da13cf5a04302add310d7"
server_key_iv: "3f36a38cb34ee20cd6d73bbb5d3dd44e"
server_key_version: "encryptionkey@0"
I20260924 03:09:47.001869 7687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:47.020437 7687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:47.024879 7687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:47.025207 7687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:47.026189 7687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:47.027565 7687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:47.027694 7687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:47.027837 7687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:47.027930 7687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:47.063549 7687 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37999
I20260924 03:09:47.063570 7814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37999 every 8 connection(s)
I20260924 03:09:47.065214 7687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:09:47.072258 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 7687
Sep 24 03:09:47 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219386, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 24 03:09:47 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219386, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 03:09:47.124220 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.068944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34973 (local address 127.0.97.3:46381)
0924 03:09:47.080687 (+ 11743us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:47.080708 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:47.090536 (+ 9828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:47.091956 (+ 1420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:47.091975 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:47.093605 (+ 1630us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:47.094795 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:47.094830 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.097804 (+ 2974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.097813 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:47.101654 (+ 3841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:47.101679 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.104379 (+ 2700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.104390 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:47.104845 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.108539 (+ 3694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:47.108583 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:47.112632 (+ 4049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:47.118700 (+ 6068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:47.118714 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:47.118741 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:47.119527 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:47.120093 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:47.120101 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:47.120108 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:47.120397 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:47.121150 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:47.121160 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:47.121631 (+ 471us) client_negotiation.cc:769] Sending connection context
0924 03:09:47.121995 (+ 364us) client_negotiation.cc:240] Negotiation successful
0924 03:09:47.122613 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11614,"spinlock_wait_cycles":63872,"thread_start_us":11524,"threads_started":1}
I20260924 03:09:47.126386 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.069069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46381 (local address 127.0.97.61:34973)
0924 03:09:47.081327 (+ 12258us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:47.081333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:47.081357 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:09:47.091227 (+ 9870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:47.091236 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:47.091380 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:47.092562 (+ 1182us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:47.095119 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.097529 (+ 2410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.102595 (+ 5066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.103694 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.103866 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.112931 (+ 9065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:47.112938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:47.112942 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:47.113005 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:47.118495 (+ 5490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:47.119759 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:47.119767 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:47.119771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:47.119925 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:47.120585 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:47.120593 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:47.120599 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:47.120977 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:47.125086 (+ 4109us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:47.125227 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 03:09:47.125684 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12109,"thread_start_us":12022,"threads_started":1}
I20260924 03:09:47.128660 7817 heartbeater.cc:347] Connected to a master server at 127.0.97.61:34973
I20260924 03:09:47.129087 7817 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:47.129462 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.068723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58443 (local address 127.0.97.62:34603)
0924 03:09:47.088151 (+ 19428us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:47.088161 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:47.088194 (+ 33us) server_negotiation.cc:407] Connection header received
0924 03:09:47.105773 (+ 17579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:47.105782 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:47.105933 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:47.106056 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:47.109265 (+ 3209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.110896 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.113768 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.114495 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.114591 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.120826 (+ 6235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:47.120833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:47.120837 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:47.120873 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:47.124574 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:47.125632 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:47.125638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:47.125642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:47.125794 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:47.127027 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:47.127034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:47.127040 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:47.127696 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:47.127846 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:47.128866 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0924 03:09:47.129097 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19208,"thread_start_us":13023,"threads_started":1}
I20260924 03:09:47.130055 7817 heartbeater.cc:511] Master 127.0.97.61:34973 requested a full tablet report, sending...
I20260924 03:09:47.131971 7163 ts_manager.cc:194] Registered new tserver with Master: 8a3d53cd8e5c47cbb4da460f4198db40 (127.0.97.3:37999)
I20260924 03:09:47.132453 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.068879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.3:58443)
0924 03:09:47.078505 (+ 9626us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:47.078580 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:47.105501 (+ 26921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:47.106442 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:47.106593 (+ 151us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:47.108226 (+ 1633us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:47.108814 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:47.109015 (+ 201us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.111163 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.111172 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:47.113608 (+ 2436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:47.113620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.114862 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.114868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:47.115492 (+ 624us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.116061 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:47.116085 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:47.119862 (+ 3777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:47.124763 (+ 4901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:47.124771 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:47.124775 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:47.125256 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:47.126663 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:47.126668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:47.126672 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:47.126860 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:47.127911 (+ 1051us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:47.127919 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:47.128102 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 03:09:47.128244 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 03:09:47.128583 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9148,"thread_start_us":11828,"threads_started":1}
I20260924 03:09:47.134016 7815 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34603
I20260924 03:09:47.134145 7815 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:47.134696 7815 heartbeater.cc:511] Master 127.0.97.62:34603 requested a full tablet report, sending...
I20260924 03:09:47.135703 6563 ts_manager.cc:194] Re-registered known tserver with Master: 8a3d53cd8e5c47cbb4da460f4198db40 (127.0.97.3:37999)
Sep 24 03:09:47 dist-test-slave-5l98 krb5kdc[6518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219386, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260924 03:09:47.136831 6563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58443
I20260924 03:09:47.137557 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.114054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33542 (local address 127.0.97.62:34603)
0924 03:09:47.116706 (+ 2652us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:47.116712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:47.117470 (+ 758us) server_negotiation.cc:407] Connection header received
0924 03:09:47.117777 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:47.117785 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:47.117883 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:47.117990 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:47.121058 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.122807 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.125284 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.126194 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.126315 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.130899 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:47.130906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:47.130923 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:47.130957 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:47.134386 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:47.135221 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:47.135226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:47.135230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:47.135367 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:47.135833 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:47.135837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:47.135840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:47.136198 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:47.136418 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:47.136983 (+ 565us) server_negotiation.cc:299] Negotiation successful
0924 03:09:47.137192 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2538,"thread_start_us":1846,"threads_started":1}
I20260924 03:09:47.144743 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.081961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57989 (local address 127.0.97.60:32857)
0924 03:09:47.108177 (+ 26216us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:47.108187 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:47.111755 (+ 3568us) server_negotiation.cc:407] Connection header received
0924 03:09:47.111894 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:47.111900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:47.111988 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:47.112104 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:47.113439 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.114932 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.130373 (+ 15441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.131071 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.131168 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.137040 (+ 5872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:47.137047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:47.137051 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:47.137091 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:47.141448 (+ 4357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:47.142347 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:47.142352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:47.142359 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:47.142544 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:47.143038 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:47.143044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:47.143051 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:47.143398 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:47.143565 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:47.144131 (+ 566us) server_negotiation.cc:299] Negotiation successful
0924 03:09:47.144352 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26039,"thread_start_us":26935,"threads_started":1}
I20260924 03:09:47.145661 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.081958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32857 (local address 127.0.97.3:57989)
0924 03:09:47.111655 (+ 29697us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:47.111683 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:47.111801 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:47.112168 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:47.112173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:47.112857 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:47.113218 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:47.113230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.115196 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.115201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:47.129403 (+ 14202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:47.129416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.131463 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.131472 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:47.131684 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.133786 (+ 2102us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:47.133814 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:47.136804 (+ 2990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:47.141679 (+ 4875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:47.141683 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:47.141687 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:47.142204 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:47.142701 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:47.142707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:47.142712 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:47.142889 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:47.143667 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:47.143673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:47.143837 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 03:09:47.144939 (+ 1102us) client_negotiation.cc:240] Negotiation successful
0924 03:09:47.145285 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29545,"thread_start_us":29327,"threads_started":1}
I20260924 03:09:47.146761 7816 heartbeater.cc:347] Connected to a master server at 127.0.97.60:32857
I20260924 03:09:47.146896 7816 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:47.147339 7816 heartbeater.cc:511] Master 127.0.97.60:32857 requested a full tablet report, sending...
I20260924 03:09:47.149027 7350 ts_manager.cc:194] Registered new tserver with Master: 8a3d53cd8e5c47cbb4da460f4198db40 (127.0.97.3:37999)
I20260924 03:09:47.149456 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.117477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55606 (local address 127.0.97.61:34973)
0924 03:09:47.126951 (+ 9474us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:47.126958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:47.130689 (+ 3731us) server_negotiation.cc:407] Connection header received
0924 03:09:47.130951 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:47.130956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:47.131046 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:47.131148 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:47.134117 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.135210 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.137224 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.138336 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.138466 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.142337 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:47.142345 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:47.142351 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:47.142393 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:47.145850 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:47.146656 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:47.146661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:47.146664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:47.146810 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:47.147263 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:47.147268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:47.147271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:47.148022 (+ 751us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:47.148257 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:47.148862 (+ 605us) server_negotiation.cc:299] Negotiation successful
0924 03:09:47.149094 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9346,"thread_start_us":9757,"threads_started":1}
I20260924 03:09:47.145618 6563 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 03:09:47.156726 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.133617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44094 (local address 127.0.97.60:32857)
0924 03:09:47.140109 (+ 6492us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:47.140118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:47.140139 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:09:47.140199 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:47.140207 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:47.140283 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:47.140405 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:47.142069 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.143342 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.146057 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.146754 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.146955 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.150976 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:47.150982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:47.150986 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:47.151021 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:47.153591 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:47.154378 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:47.154381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:47.154384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:47.154566 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:47.155072 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:47.155080 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:47.155087 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:47.155483 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:47.155689 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:47.156066 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 03:09:47.156247 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6343,"thread_start_us":6207,"threads_started":1}
W20260924 03:09:47.156834 6563 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:47.201476 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.179826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37927 (local address 127.0.0.1:41100)
0924 03:09:47.190302 (+ 10476us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:47.190330 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:47.191010 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:47.192879 (+ 1869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:47.192882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:47.192906 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:47.193360 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:47.193370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.195213 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.195219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:47.198560 (+ 3341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:47.198575 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.200659 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.200665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:47.200908 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.200963 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 03:09:47.201046 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 03:09:47.201165 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9802,"mutex_wait_us":23,"thread_start_us":18793,"threads_started":1}
I20260924 03:09:47.202288 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.180253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41100 (local address 127.0.97.1:37927)
0924 03:09:47.192011 (+ 11758us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:47.192022 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:47.192059 (+ 37us) server_negotiation.cc:407] Connection header received
0924 03:09:47.192193 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:47.192206 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:47.192594 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:47.192878 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:47.193660 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.195062 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.198733 (+ 3671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.200321 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.200514 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.200741 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:47.201796 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0924 03:09:47.201985 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11571,"thread_start_us":12843,"threads_started":1}
I20260924 03:09:47.205192 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.180192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37999 (local address 127.0.0.1:57844)
0924 03:09:47.193462 (+ 13270us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:47.193485 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:47.193621 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:47.194310 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:47.194317 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:47.194349 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:47.194838 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:47.194853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.198719 (+ 3866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.198723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:47.200889 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:47.200902 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.204325 (+ 3423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.204331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:47.204514 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.204556 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 03:09:47.204640 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 03:09:47.204779 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11075,"mutex_wait_us":41,"thread_start_us":10905,"threads_started":1}
I20260924 03:09:47.206517 7463 tablet_service.cc:1467] Processing CreateTablet for tablet 183c6abeba4a4978b1c20dd26380dcf2 (DEFAULT_TABLE table=test-table [id=055f74aa6e9c43e7bb38d65707afea87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:47.207295 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.181127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43547 (local address 127.0.0.1:52606)
0924 03:09:47.197876 (+ 16749us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:47.197900 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:47.198500 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:47.199631 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:47.199636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:47.199664 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:47.200186 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:47.200198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.202132 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.202137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:47.204058 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:47.204070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.206405 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.206411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:47.206594 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.206628 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:09:47.206791 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 03:09:47.206920 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16623,"thread_start_us":9568,"threads_started":1}
I20260924 03:09:47.207405 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.180839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.0.97.2:43547)
0924 03:09:47.189249 (+ 8410us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:47.189264 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:47.198068 (+ 8804us) server_negotiation.cc:407] Connection header received
0924 03:09:47.198751 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:47.198761 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:47.199091 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:47.199311 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:47.200479 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.201998 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.204232 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.205948 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.206077 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.206259 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:47.206903 (+ 644us) server_negotiation.cc:299] Negotiation successful
0924 03:09:47.207093 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8158,"thread_start_us":7995,"threads_started":1}
I20260924 03:09:47.208029 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 183c6abeba4a4978b1c20dd26380dcf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:47.209183 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.180783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57844 (local address 127.0.97.3:37999)
0924 03:09:47.191786 (+ 11003us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:47.191797 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:47.193579 (+ 1782us) server_negotiation.cc:407] Connection header received
0924 03:09:47.193817 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:47.193826 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:47.194094 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:47.194286 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:47.195280 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.196768 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.201068 (+ 4300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.202634 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.202731 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.202870 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:47.208657 (+ 5787us) server_negotiation.cc:299] Negotiation successful
0924 03:09:47.208854 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10805,"thread_start_us":10638,"threads_started":1}
I20260924 03:09:47.211174 7609 tablet_service.cc:1467] Processing CreateTablet for tablet 183c6abeba4a4978b1c20dd26380dcf2 (DEFAULT_TABLE table=test-table [id=055f74aa6e9c43e7bb38d65707afea87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:47.212342 7609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 183c6abeba4a4978b1c20dd26380dcf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:47.213331 7750 tablet_service.cc:1467] Processing CreateTablet for tablet 183c6abeba4a4978b1c20dd26380dcf2 (DEFAULT_TABLE table=test-table [id=055f74aa6e9c43e7bb38d65707afea87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:47.214747 7750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 183c6abeba4a4978b1c20dd26380dcf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:47.225073 7852 tablet_bootstrap.cc:492] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f: Bootstrap starting.
I20260924 03:09:47.228216 7852 tablet_bootstrap.cc:654] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:47.229729 7852 log.cc:824] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:47.231716 7853 tablet_bootstrap.cc:492] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee: Bootstrap starting.
I20260924 03:09:47.233652 7855 tablet_bootstrap.cc:492] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40: Bootstrap starting.
I20260924 03:09:47.233704 7852 tablet_bootstrap.cc:492] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f: No bootstrap required, opened a new log
I20260924 03:09:47.233989 7852 ts_tablet_manager.cc:1397] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 03:09:47.235881 7853 tablet_bootstrap.cc:654] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:47.236585 7855 tablet_bootstrap.cc:654] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:47.237208 7853 log.cc:824] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:47.237627 7855 log.cc:824] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:47.240876 7853 tablet_bootstrap.cc:492] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee: No bootstrap required, opened a new log
I20260924 03:09:47.240909 7855 tablet_bootstrap.cc:492] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40: No bootstrap required, opened a new log
I20260924 03:09:47.241174 7853 ts_tablet_manager.cc:1397] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 03:09:47.241220 7855 ts_tablet_manager.cc:1397] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 03:09:47.243781 7852 raft_consensus.cc:348] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 } } peers { permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 } }
I20260924 03:09:47.244174 7852 raft_consensus.cc:374] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:47.244269 7852 raft_consensus.cc:729] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 765e21fdfb6049d48dcb457debd0e02f, State: Initialized, Role: FOLLOWER
I20260924 03:09:47.244973 7852 consensus_queue.cc:260] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 } } peers { permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 } }
I20260924 03:09:47.246102 7674 heartbeater.cc:503] Master 127.0.97.62:34603 was elected leader, sending a full tablet report...
I20260924 03:09:47.248157 7852 ts_tablet_manager.cc:1428] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260924 03:09:47.252039 7853 raft_consensus.cc:348] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 } } peers { permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 } }
I20260924 03:09:47.252516 7853 raft_consensus.cc:374] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:47.252630 7853 raft_consensus.cc:729] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00cb9dff7a9b430c9ad5224aa776edee, State: Initialized, Role: FOLLOWER
I20260924 03:09:47.253234 7853 consensus_queue.cc:260] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 } } peers { permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 } }
I20260924 03:09:47.255250 7853 ts_tablet_manager.cc:1428] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260924 03:09:47.256059 7855 raft_consensus.cc:348] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 } } peers { permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 } }
I20260924 03:09:47.257120 7855 raft_consensus.cc:374] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:47.257228 7855 raft_consensus.cc:729] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3d53cd8e5c47cbb4da460f4198db40, State: Initialized, Role: FOLLOWER
I20260924 03:09:47.258935 7855 consensus_queue.cc:260] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 } } peers { permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 } }
I20260924 03:09:47.260174 7815 heartbeater.cc:503] Master 127.0.97.62:34603 was elected leader, sending a full tablet report...
I20260924 03:09:47.260900 7855 ts_tablet_manager.cc:1428] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40: Time spent starting tablet: real 0.020s user 0.010s sys 0.004s
W20260924 03:09:47.282642 7533 tablet.cc:2369] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:09:47.358068 7818 tablet.cc:2369] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:09:47.374897 7859 raft_consensus.cc:482] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:47.375270 7859 raft_consensus.cc:504] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 } } peers { permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 } }
I20260924 03:09:47.377315 7859 leader_election.cc:288] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a3d53cd8e5c47cbb4da460f4198db40 (127.0.97.3:37999), 765e21fdfb6049d48dcb457debd0e02f (127.0.97.2:43547)
I20260924 03:09:47.377554 7860 raft_consensus.cc:482] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:47.377868 7860 raft_consensus.cc:504] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 } } peers { permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 } }
I20260924 03:09:47.380903 7860 leader_election.cc:288] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00cb9dff7a9b430c9ad5224aa776edee (127.0.97.1:37927), 765e21fdfb6049d48dcb457debd0e02f (127.0.97.2:43547)
I20260924 03:09:47.388525 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.378434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57489 (local address 127.0.97.2:43547)
0924 03:09:47.378639 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:47.378658 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:47.379663 (+ 1005us) server_negotiation.cc:407] Connection header received
0924 03:09:47.379899 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:47.379906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:47.379983 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:47.380060 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:47.381759 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.382875 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.385842 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.387125 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.387245 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.387347 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:47.388075 (+ 728us) server_negotiation.cc:299] Negotiation successful
0924 03:09:47.388224 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 03:09:47.388617 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.377655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37999 (local address 127.0.97.1:57979)
0924 03:09:47.379490 (+ 1835us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:47.379507 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:47.379698 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:47.380062 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:47.380066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:47.380103 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:47.381300 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:47.381315 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.382931 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.382936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:47.385654 (+ 2718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:47.385668 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.387532 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.387544 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:47.387806 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.387922 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 03:09:47.388025 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 03:09:47.388164 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1666,"mutex_wait_us":195,"thread_start_us":412,"threads_started":1}
I20260924 03:09:47.389353 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.378784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57979 (local address 127.0.97.3:37999)
0924 03:09:47.378976 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:47.378982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:47.379612 (+ 630us) server_negotiation.cc:407] Connection header received
0924 03:09:47.379833 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:47.379837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:47.379921 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:47.380014 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:47.381493 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.382711 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.385829 (+ 3118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.387150 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.387258 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.387355 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:47.388959 (+ 1604us) server_negotiation.cc:299] Negotiation successful
0924 03:09:47.389111 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":57}
I20260924 03:09:47.389763 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.377754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43547 (local address 127.0.97.1:57489)
0924 03:09:47.379548 (+ 1794us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:47.379570 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:47.379770 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:47.380219 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:47.380223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:47.380252 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:47.381508 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:47.381519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.383058 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.383064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:47.385654 (+ 2590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:47.385668 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.387563 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.387569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:47.387812 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.387922 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 03:09:47.389151 (+ 1229us) client_negotiation.cc:240] Negotiation successful
0924 03:09:47.389300 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1666,"mutex_wait_us":244,"thread_start_us":1513,"threads_started":1}
I20260924 03:09:47.390645 7770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "183c6abeba4a4978b1c20dd26380dcf2" candidate_uuid: "00cb9dff7a9b430c9ad5224aa776edee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" is_pre_election: true
I20260924 03:09:47.391248 7770 raft_consensus.cc:2454] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00cb9dff7a9b430c9ad5224aa776edee in term 0.
I20260924 03:09:47.391784 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.380316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37927 (local address 127.0.97.3:36265)
0924 03:09:47.381187 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:47.381207 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:47.381325 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:47.383174 (+ 1849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:47.383180 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:47.383207 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:47.383378 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:47.383392 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.385328 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.385336 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:47.388077 (+ 2741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:47.388097 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.390654 (+ 2557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.390661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:47.390923 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.391052 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 03:09:47.391150 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 03:09:47.391309 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":1}
I20260924 03:09:47.391633 7629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "183c6abeba4a4978b1c20dd26380dcf2" candidate_uuid: "00cb9dff7a9b430c9ad5224aa776edee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "765e21fdfb6049d48dcb457debd0e02f" is_pre_election: true
I20260924 03:09:47.392159 7416 leader_election.cc:302] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00cb9dff7a9b430c9ad5224aa776edee, 8a3d53cd8e5c47cbb4da460f4198db40; no voters:
I20260924 03:09:47.392212 7629 raft_consensus.cc:2454] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00cb9dff7a9b430c9ad5224aa776edee in term 0.
I20260924 03:09:47.392738 7859 raft_consensus.cc:2781] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:47.392942 7859 raft_consensus.cc:482] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:47.393124 7859 raft_consensus.cc:3037] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:47.393178 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.380375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36265 (local address 127.0.97.1:37927)
0924 03:09:47.381603 (+ 1228us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:47.381619 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:47.381635 (+ 16us) server_negotiation.cc:407] Connection header received
0924 03:09:47.381684 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:47.381688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:47.381796 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:47.381911 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:47.383553 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.385106 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.388333 (+ 3227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.390457 (+ 2124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.390624 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.390713 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:47.392078 (+ 1365us) server_negotiation.cc:299] Negotiation successful
0924 03:09:47.392222 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1069}
I20260924 03:09:47.394006 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "183c6abeba4a4978b1c20dd26380dcf2" candidate_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00cb9dff7a9b430c9ad5224aa776edee" is_pre_election: true
I20260924 03:09:47.396109 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.382628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43547 (local address 127.0.97.3:46191)
0924 03:09:47.382859 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:47.382873 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:47.382952 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:47.390232 (+ 7280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:47.390238 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:47.390277 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:47.390448 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:47.390476 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.392144 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.392152 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:47.393905 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:47.393918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.395339 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.395344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:47.395552 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.395580 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 03:09:47.395727 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 03:09:47.395846 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260924 03:09:47.395992 7859 raft_consensus.cc:504] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 } } peers { permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 } }
I20260924 03:09:47.396283 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.383298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46191 (local address 127.0.97.2:43547)
0924 03:09:47.388792 (+ 5494us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:47.388797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:47.388813 (+ 16us) server_negotiation.cc:407] Connection header received
0924 03:09:47.388890 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:47.388895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:47.388972 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:47.389054 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:47.390707 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.391983 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.394253 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.395156 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.395331 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.395434 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:47.395787 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 03:09:47.395966 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5389,"thread_start_us":6286,"threads_started":1}
I20260924 03:09:47.396762 7483 raft_consensus.cc:2379] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a3d53cd8e5c47cbb4da460f4198db40 in current term 1: Already voted for candidate 00cb9dff7a9b430c9ad5224aa776edee in this term.
I20260924 03:09:47.396862 7859 leader_election.cc:288] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [CANDIDATE]: Term 1 election: Requested vote from peers 8a3d53cd8e5c47cbb4da460f4198db40 (127.0.97.3:37999), 765e21fdfb6049d48dcb457debd0e02f (127.0.97.2:43547)
I20260924 03:09:47.396926 7629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "183c6abeba4a4978b1c20dd26380dcf2" candidate_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "765e21fdfb6049d48dcb457debd0e02f" is_pre_election: true
I20260924 03:09:47.397130 7629 raft_consensus.cc:2454] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a3d53cd8e5c47cbb4da460f4198db40 in term 0.
I20260924 03:09:47.397284 7770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "183c6abeba4a4978b1c20dd26380dcf2" candidate_uuid: "00cb9dff7a9b430c9ad5224aa776edee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a3d53cd8e5c47cbb4da460f4198db40"
I20260924 03:09:47.397454 7629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "183c6abeba4a4978b1c20dd26380dcf2" candidate_uuid: "00cb9dff7a9b430c9ad5224aa776edee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "765e21fdfb6049d48dcb457debd0e02f"
I20260924 03:09:47.397534 7770 raft_consensus.cc:3037] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:47.397629 7629 raft_consensus.cc:3037] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:47.397578 7706 leader_election.cc:302] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 765e21fdfb6049d48dcb457debd0e02f, 8a3d53cd8e5c47cbb4da460f4198db40; no voters:
I20260924 03:09:47.400360 7629 raft_consensus.cc:2454] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00cb9dff7a9b430c9ad5224aa776edee in term 1.
I20260924 03:09:47.400785 7419 leader_election.cc:302] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00cb9dff7a9b430c9ad5224aa776edee, 765e21fdfb6049d48dcb457debd0e02f; no voters:
I20260924 03:09:47.400875 7770 raft_consensus.cc:2454] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00cb9dff7a9b430c9ad5224aa776edee in term 1.
I20260924 03:09:47.401111 7859 raft_consensus.cc:2781] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:47.401212 7860 raft_consensus.cc:2741] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260924 03:09:47.401613 7859 raft_consensus.cc:686] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [term 1 LEADER]: Becoming Leader. State: Replica: 00cb9dff7a9b430c9ad5224aa776edee, State: Running, Role: LEADER
I20260924 03:09:47.402199 7859 consensus_queue.cc:237] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 } } peers { permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 } }
I20260924 03:09:47.407871 6562 catalog_manager.cc:5495] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee reported cstate change: term changed from 0 to 1, leader changed from <none> to 00cb9dff7a9b430c9ad5224aa776edee (127.0.97.1). New cstate: current_term: 1 leader_uuid: "00cb9dff7a9b430c9ad5224aa776edee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 } health_report { overall_health: UNKNOWN } } }
W20260924 03:09:47.428197 7677 tablet.cc:2369] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:09:47.441725 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.436150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41112 (local address 127.0.97.1:37927)
0924 03:09:47.436368 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:47.436375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:47.436462 (+ 87us) server_negotiation.cc:407] Connection header received
0924 03:09:47.436744 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:47.436748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:47.436839 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:47.437014 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:47.437361 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.438158 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.439445 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.439865 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.440001 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.440553 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:47.441109 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:47.441283 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:47.441334 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 03:09:47.441424 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 03:09:47.450069 7629 raft_consensus.cc:1261] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f [term 1 FOLLOWER]: Refusing update from remote peer 00cb9dff7a9b430c9ad5224aa776edee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:47.450109 7770 raft_consensus.cc:1261] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 1 FOLLOWER]: Refusing update from remote peer 00cb9dff7a9b430c9ad5224aa776edee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:47.451004 7859 consensus_queue.cc:1035] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [LEADER]: Connected to new peer: Peer: permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:47.451309 7864 consensus_queue.cc:1035] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:47.476297 7869 mvcc.cc:205] Tried to move back new op lower bound from 7332738610985226240 to 7332738610803994624. Current Snapshot: MvccSnapshot[applied={T|T < 7332738610985226240}]
I20260924 03:09:47.495280 6562 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:47.495610 6562 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:47.502568 6562 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a226d7c36f749c78880160588ba0ba1: Sending DeleteTablet for 3 replicas of tablet 183c6abeba4a4978b1c20dd26380dcf2
I20260924 03:09:47.503762 7463 tablet_service.cc:1514] Processing DeleteTablet for tablet 183c6abeba4a4978b1c20dd26380dcf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:47 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41100
I20260924 03:09:47.504347 7750 tablet_service.cc:1514] Processing DeleteTablet for tablet 183c6abeba4a4978b1c20dd26380dcf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:47 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57844
I20260924 03:09:47.504921 7876 tablet_replica.cc:330] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee: stopping tablet replica
I20260924 03:09:47.505070 7609 tablet_service.cc:1514] Processing DeleteTablet for tablet 183c6abeba4a4978b1c20dd26380dcf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:47 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52606
I20260924 03:09:47.505365 7876 raft_consensus.cc:2229] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:09:47.506412 7876 raft_consensus.cc:2258] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:47.513063 7877 tablet_replica.cc:330] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40: stopping tablet replica
I20260924 03:09:47.513521 7877 raft_consensus.cc:2229] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:47.514092 7877 raft_consensus.cc:2258] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:47.515789 7876 ts_tablet_manager.cc:1905] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:47.516737 7877 ts_tablet_manager.cc:1905] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:47.519243 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 6531
I20260924 03:09:47.520463 6593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:47.523398 7876 ts_tablet_manager.cc:1918] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:47.523684 7876 log.cc:1196] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/183c6abeba4a4978b1c20dd26380dcf2
I20260924 03:09:47.524340 7876 ts_tablet_manager.cc:1939] T 183c6abeba4a4978b1c20dd26380dcf2 P 00cb9dff7a9b430c9ad5224aa776edee: Deleting consensus metadata
I20260924 03:09:47.529109 7878 tablet_replica.cc:330] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f: stopping tablet replica
I20260924 03:09:47.529587 7878 raft_consensus.cc:2229] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:47.530104 7878 raft_consensus.cc:2258] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:47.532625 7878 ts_tablet_manager.cc:1905] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:47.534058 7877 ts_tablet_manager.cc:1918] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:47.534225 7877 log.cc:1196] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/183c6abeba4a4978b1c20dd26380dcf2
I20260924 03:09:47.534715 7877 ts_tablet_manager.cc:1939] T 183c6abeba4a4978b1c20dd26380dcf2 P 8a3d53cd8e5c47cbb4da460f4198db40: Deleting consensus metadata
I20260924 03:09:47.539490 7878 ts_tablet_manager.cc:1918] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:47.539650 7878 log.cc:1196] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/183c6abeba4a4978b1c20dd26380dcf2
I20260924 03:09:47.540216 7878 ts_tablet_manager.cc:1939] T 183c6abeba4a4978b1c20dd26380dcf2 P 765e21fdfb6049d48dcb457debd0e02f: Deleting consensus metadata
I20260924 03:09:47.548974 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.507374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55612 (local address 127.0.97.61:34973)
0924 03:09:47.507608 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:47.507617 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:47.517287 (+ 9670us) server_negotiation.cc:407] Connection header received
0924 03:09:47.519147 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:47.519156 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:47.519268 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:47.519359 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:47.531662 (+ 12303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.533278 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.536402 (+ 3124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.537049 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.537158 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.541151 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:47.541159 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:47.541164 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:47.541206 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:47.545597 (+ 4391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:47.546600 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:47.546605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:47.546609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:47.546763 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:47.547244 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:47.547249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:47.547252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:47.547688 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:47.548018 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:47.548314 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 03:09:47.548567 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:09:47.566304 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.517949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44104 (local address 127.0.97.60:32857)
0924 03:09:47.518185 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:47.518194 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:47.549207 (+ 31013us) server_negotiation.cc:407] Connection header received
0924 03:09:47.549595 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:47.549602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:47.549690 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:47.549765 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:47.551761 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.552975 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.554894 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.555582 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.555734 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.559239 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:47.559245 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:47.559251 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:47.559292 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:47.562973 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:47.563849 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:47.563856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:47.563860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:47.564016 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:47.564587 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:47.564592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:47.564596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:47.564955 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:47.565204 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:47.565562 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 03:09:47.565789 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 03:09:47.663045 6547 catalog_manager.cc:4841] TS 8a3d53cd8e5c47cbb4da460f4198db40 (127.0.97.3:37999): tablet 183c6abeba4a4978b1c20dd26380dcf2 (table test-table [id=055f74aa6e9c43e7bb38d65707afea87]) successfully deleted
I20260924 03:09:47.664448 6550 catalog_manager.cc:4841] TS 765e21fdfb6049d48dcb457debd0e02f (127.0.97.2:43547): tablet 183c6abeba4a4978b1c20dd26380dcf2 (table test-table [id=055f74aa6e9c43e7bb38d65707afea87]) successfully deleted
I20260924 03:09:47.665226 6549 catalog_manager.cc:4841] TS 00cb9dff7a9b430c9ad5224aa776edee (127.0.97.1:37927): tablet 183c6abeba4a4978b1c20dd26380dcf2 (table test-table [id=055f74aa6e9c43e7bb38d65707afea87]) successfully deleted
I20260924 03:09:47.665352 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 6531
I20260924 03:09:47.704979 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.689223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44116 (local address 127.0.97.60:32857)
0924 03:09:47.689403 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:47.689407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:47.689646 (+ 239us) server_negotiation.cc:407] Connection header received
0924 03:09:47.689856 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:47.689861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:47.689933 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:47.690041 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:47.691877 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.692779 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.694091 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.694513 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.694655 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.698613 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:47.698624 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:47.698628 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:47.698669 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:47.701676 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:47.702386 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:47.702392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:47.702396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:47.702550 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:47.702984 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:47.702991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:47.703034 (+ 43us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:47.703472 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:47.703731 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:47.704148 (+ 417us) server_negotiation.cc:299] Negotiation successful
0924 03:09:47.704615 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:09:47.705694 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:47.687357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55614 (local address 127.0.97.61:34973)
0924 03:09:47.687560 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:47.687566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:47.688452 (+ 886us) server_negotiation.cc:407] Connection header received
0924 03:09:47.688680 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:47.688686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:47.688769 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:47.688859 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:47.691024 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.692136 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.694121 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:47.694756 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:47.694888 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:47.698583 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:47.698589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:47.698595 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:47.698638 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:47.702340 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:47.703047 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:47.703053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:47.703056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:47.703239 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:47.703653 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:47.703659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:47.703662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:47.704079 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:47.704369 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:47.705239 (+ 870us) server_negotiation.cc:299] Negotiation successful
0924 03:09:47.705451 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 03:09:48.483049 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:48.482106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.2:40171)
0924 03:09:48.482585 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:48.482733 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":229,"threads_started":1}
W20260924 03:09:48.483512 7674 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34603 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
I20260924 03:09:48.485169 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:48.484268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.1:45871)
0924 03:09:48.484860 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:48.484993 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":390,"threads_started":1}
W20260924 03:09:48.485522 7528 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34603 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
I20260924 03:09:48.491842 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:48.490928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.3:46831)
0924 03:09:48.491504 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:48.491658 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":261,"threads_started":1}
W20260924 03:09:48.492318 7815 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34603 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
I20260924 03:09:49.003542 7890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a226d7c36f749c78880160588ba0ba1)
I20260924 03:09:49.003850 7891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a226d7c36f749c78880160588ba0ba1)
I20260924 03:09:49.003873 7890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: false } }
I20260924 03:09:49.004171 7891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: false } }
I20260924 03:09:49.005615 7890 leader_election.cc:288] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a226d7c36f749c78880160588ba0ba1 (127.0.97.62:34603), 33b19f8aecbc4da1b1150ad5360242af (127.0.97.60:32857)
I20260924 03:09:49.006879 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.005667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.0.1:33980)
0924 03:09:49.006238 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.006415 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":255,"threads_started":1}
I20260924 03:09:49.006845 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.005720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.0.1:33988)
0924 03:09:49.006306 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.006490 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":259,"threads_started":1}
I20260924 03:09:49.007628 7891 leader_election.cc:288] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a226d7c36f749c78880160588ba0ba1 (127.0.97.62:34603), 782bd86c1d32482598fdc6ed60cc391b (127.0.97.61:34973)
W20260924 03:09:49.008123 7147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
W20260924 03:09:49.008255 7334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
I20260924 03:09:49.010303 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.009755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.0.1:33992)
0924 03:09:49.009932 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.010066 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 03:09:49.010648 7147 leader_election.cc:334] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a226d7c36f749c78880160588ba0ba1 (127.0.97.62:34603): Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
I20260924 03:09:49.011359 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.010647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.0.1:33998)
0924 03:09:49.010956 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.011122 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260924 03:09:49.011631 7334 leader_election.cc:334] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a226d7c36f749c78880160588ba0ba1 (127.0.97.62:34603): Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
I20260924 03:09:49.015693 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.006379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32857 (local address 127.0.0.1:44124)
0924 03:09:49.007242 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.007259 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:49.007395 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:49.008831 (+ 1436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:49.008859 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:49.008911 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:49.009322 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:49.009360 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.011367 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.011376 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:49.013397 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:49.013412 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.014811 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.014817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:49.015008 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:49.015034 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 03:09:49.015158 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 03:09:49.015324 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":1425,"threads_started":1}
I20260924 03:09:49.015702 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.007224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44124 (local address 127.0.97.60:32857)
0924 03:09:49.008372 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:49.008379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:49.008400 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:09:49.008467 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:49.008472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:49.008585 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:49.009545 (+ 960us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:49.009777 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.011183 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.013570 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.014628 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.014786 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:49.014907 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:49.015180 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 03:09:49.015316 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":942,"thread_start_us":805,"threads_started":1}
I20260924 03:09:49.016389 7360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "782bd86c1d32482598fdc6ed60cc391b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "33b19f8aecbc4da1b1150ad5360242af" is_pre_election: true
I20260924 03:09:49.016834 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.006904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34973 (local address 127.0.0.1:55628)
0924 03:09:49.007316 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.007342 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:49.007501 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:49.011298 (+ 3797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:49.011303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:49.011333 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:49.011675 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:49.011685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.013098 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.013104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:49.014746 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:49.014758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.016097 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.016103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:49.016252 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:49.016276 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 03:09:49.016333 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 03:09:49.016438 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":18,"thread_start_us":1732,"threads_started":1}
I20260924 03:09:49.016808 7360 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 782bd86c1d32482598fdc6ed60cc391b in term 1.
I20260924 03:09:49.017297 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.007490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55628 (local address 127.0.97.61:34973)
0924 03:09:49.010975 (+ 3485us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:49.010981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:49.011028 (+ 47us) server_negotiation.cc:407] Connection header received
0924 03:09:49.011089 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:49.011094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:49.011157 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:49.011275 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:49.011892 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.012927 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.014909 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.015869 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.016633 (+ 764us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:49.016735 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:49.016790 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 03:09:49.016935 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3350,"thread_start_us":4292,"threads_started":1}
I20260924 03:09:49.017475 7147 leader_election.cc:302] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33b19f8aecbc4da1b1150ad5360242af, 782bd86c1d32482598fdc6ed60cc391b; no voters: 0a226d7c36f749c78880160588ba0ba1
I20260924 03:09:49.017916 7890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 03:09:49.018082 7890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a226d7c36f749c78880160588ba0ba1)
I20260924 03:09:49.018080 7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33b19f8aecbc4da1b1150ad5360242af" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "782bd86c1d32482598fdc6ed60cc391b" is_pre_election: true
I20260924 03:09:49.018249 7890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:09:49.020462 7890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: false } }
I20260924 03:09:49.021013 7173 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 33b19f8aecbc4da1b1150ad5360242af in current term 2: Already voted for candidate 782bd86c1d32482598fdc6ed60cc391b in this term.
I20260924 03:09:49.021270 7890 leader_election.cc:288] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [CANDIDATE]: Term 2 election: Requested vote from peers 0a226d7c36f749c78880160588ba0ba1 (127.0.97.62:34603), 33b19f8aecbc4da1b1150ad5360242af (127.0.97.60:32857)
I20260924 03:09:49.021790 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.021395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.0.1:34012)
0924 03:09:49.021533 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.021623 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 03:09:49.022030 7360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "782bd86c1d32482598fdc6ed60cc391b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "33b19f8aecbc4da1b1150ad5360242af"
I20260924 03:09:49.022032 7335 leader_election.cc:302] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33b19f8aecbc4da1b1150ad5360242af; no voters: 0a226d7c36f749c78880160588ba0ba1, 782bd86c1d32482598fdc6ed60cc391b
I20260924 03:09:49.022289 7360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:09:49.024636 7360 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 782bd86c1d32482598fdc6ed60cc391b in term 2.
I20260924 03:09:49.024957 7891 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:09:49.025149 7147 leader_election.cc:302] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33b19f8aecbc4da1b1150ad5360242af, 782bd86c1d32482598fdc6ed60cc391b; no voters:
I20260924 03:09:49.025451 7890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:09:49.025961 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.025587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.0.1:34024)
0924 03:09:49.025732 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.025828 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 03:09:49.026144 7890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [term 2 LEADER]: Becoming Leader. State: Replica: 782bd86c1d32482598fdc6ed60cc391b, State: Running, Role: LEADER
W20260924 03:09:49.026319 7147 leader_election.cc:334] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0a226d7c36f749c78880160588ba0ba1 (127.0.97.62:34603): Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
I20260924 03:09:49.026590 7890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: false } }
I20260924 03:09:49.028441 7900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 782bd86c1d32482598fdc6ed60cc391b. Latest consensus state: current_term: 2 leader_uuid: "782bd86c1d32482598fdc6ed60cc391b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: false } } }
I20260924 03:09:49.028919 7900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:49.029481 7902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:09:49.033061 7902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:09:49.033869 7902 catalog_manager.cc:1259] Loaded cluster ID: adc5298acf804509a20dfe0c10861d58
I20260924 03:09:49.033982 7902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:09:49.035123 7902 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:09:49.035910 7902 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: Loaded TSK: 0
I20260924 03:09:49.036844 7902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:09:49.059304 7163 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:09:49.067975 7163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:49.072362 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.071918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.0.1:34044)
0924 03:09:49.072102 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.072197 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 03:09:49.072867 7360 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [term 2 FOLLOWER]: Refusing update from remote peer 782bd86c1d32482598fdc6ed60cc391b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 03:09:49.073491 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.073143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.0.1:34056)
0924 03:09:49.073292 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.073374 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 03:09:49.073736 7147 consensus_peers.cc:487] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b -> Peer 0a226d7c36f749c78880160588ba0ba1 (127.0.97.62:34603): Couldn't send request to peer 0a226d7c36f749c78880160588ba0ba1. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 03:09:49.073742 7890 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [LEADER]: Connected to new peer: Peer: permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 03:09:49.077193 7891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 782bd86c1d32482598fdc6ed60cc391b. Latest consensus state: current_term: 2 leader_uuid: "782bd86c1d32482598fdc6ed60cc391b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: false } } }
I20260924 03:09:49.077543 7891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:09:49.079558 7900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "782bd86c1d32482598fdc6ed60cc391b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: false } } }
I20260924 03:09:49.079833 7900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:49.080408 7891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "782bd86c1d32482598fdc6ed60cc391b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0a226d7c36f749c78880160588ba0ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34603 } } peers { permanent_uuid: "782bd86c1d32482598fdc6ed60cc391b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "33b19f8aecbc4da1b1150ad5360242af" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32857 } attrs { promote: false } } }
I20260924 03:09:49.080862 7891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b19f8aecbc4da1b1150ad5360242af [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:09:49.096143 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.088626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37999 (local address 127.0.0.1:57846)
0924 03:09:49.089173 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.089193 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:49.089310 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:49.089929 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:49.089941 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:49.089970 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:49.090180 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:49.090190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.091679 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.091687 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:49.093583 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:49.093595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.095234 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.095239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:49.095483 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:49.095522 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 03:09:49.095621 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 03:09:49.095778 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":27}
I20260924 03:09:49.097157 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.088887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57846 (local address 127.0.97.3:37999)
0924 03:09:49.089591 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:49.089602 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:49.089618 (+ 16us) server_negotiation.cc:407] Connection header received
0924 03:09:49.089671 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:49.089675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:49.089726 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:49.090325 (+ 599us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:49.090514 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.091549 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.093797 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.095082 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.095224 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:49.095328 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:49.096589 (+ 1261us) server_negotiation.cc:299] Negotiation successful
0924 03:09:49.096737 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":515,"threads_started":1}
I20260924 03:09:49.097932 7750 tablet_service.cc:1467] Processing CreateTablet for tablet 92580e541f6e400c94c2cae00a159d83 (DEFAULT_TABLE table=test-table [id=0bf5bfe90d2843a3b52abac5266d6215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:49.098429 7750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92580e541f6e400c94c2cae00a159d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:49.098893 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.089581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37927 (local address 127.0.0.1:41120)
0924 03:09:49.090188 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.090204 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:49.090299 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:49.092338 (+ 2039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:49.092346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:49.092383 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:49.092911 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:49.092920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.094139 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.094147 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:49.096042 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:49.096055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.098011 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.098017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:49.098190 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:49.098222 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 03:09:49.098317 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 03:09:49.098449 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":19,"thread_start_us":313,"threads_started":1}
I20260924 03:09:49.098930 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.089979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41120 (local address 127.0.97.1:37927)
0924 03:09:49.091995 (+ 2016us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:49.092002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:49.092019 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:09:49.092081 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:49.092086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:49.092166 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:49.092271 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:49.093080 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.093996 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.096210 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.097655 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.097747 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:49.097835 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:49.098363 (+ 528us) server_negotiation.cc:299] Negotiation successful
0924 03:09:49.098645 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1900,"thread_start_us":1796,"threads_started":1}
I20260924 03:09:49.099344 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.089148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43547 (local address 127.0.0.1:52612)
0924 03:09:49.089401 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.089419 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:49.089547 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:49.093197 (+ 3650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:49.093203 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:49.093229 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:49.093560 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:49.093571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.094955 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.094960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:49.097044 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:49.097057 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.098695 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.098700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:49.098906 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:49.098948 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 03:09:49.099025 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 03:09:49.099152 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":21}
I20260924 03:09:49.100147 7463 tablet_service.cc:1467] Processing CreateTablet for tablet 92580e541f6e400c94c2cae00a159d83 (DEFAULT_TABLE table=test-table [id=0bf5bfe90d2843a3b52abac5266d6215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:49.100273 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.089241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.0.97.2:43547)
0924 03:09:49.092905 (+ 3664us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:49.092911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:49.092929 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:09:49.092993 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:49.092998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:49.093063 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:49.093161 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:49.093720 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.094793 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.097260 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.098460 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.099697 (+ 1237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:49.099771 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:49.099852 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 03:09:49.099980 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3549,"thread_start_us":3455,"threads_started":1}
I20260924 03:09:49.100651 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92580e541f6e400c94c2cae00a159d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:49.100965 7609 tablet_service.cc:1467] Processing CreateTablet for tablet 92580e541f6e400c94c2cae00a159d83 (DEFAULT_TABLE table=test-table [id=0bf5bfe90d2843a3b52abac5266d6215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:49.101418 7609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92580e541f6e400c94c2cae00a159d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:49.107913 7913 tablet_bootstrap.cc:492] T 92580e541f6e400c94c2cae00a159d83 P 765e21fdfb6049d48dcb457debd0e02f: Bootstrap starting.
I20260924 03:09:49.109207 7912 tablet_bootstrap.cc:492] T 92580e541f6e400c94c2cae00a159d83 P 00cb9dff7a9b430c9ad5224aa776edee: Bootstrap starting.
I20260924 03:09:49.110693 7913 tablet_bootstrap.cc:654] T 92580e541f6e400c94c2cae00a159d83 P 765e21fdfb6049d48dcb457debd0e02f: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:49.110934 7912 tablet_bootstrap.cc:654] T 92580e541f6e400c94c2cae00a159d83 P 00cb9dff7a9b430c9ad5224aa776edee: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:49.113627 7912 tablet_bootstrap.cc:492] T 92580e541f6e400c94c2cae00a159d83 P 00cb9dff7a9b430c9ad5224aa776edee: No bootstrap required, opened a new log
I20260924 03:09:49.113817 7912 ts_tablet_manager.cc:1397] T 92580e541f6e400c94c2cae00a159d83 P 00cb9dff7a9b430c9ad5224aa776edee: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 03:09:49.114877 7912 raft_consensus.cc:348] T 92580e541f6e400c94c2cae00a159d83 P 00cb9dff7a9b430c9ad5224aa776edee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 } } peers { permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 } }
I20260924 03:09:49.115206 7912 raft_consensus.cc:374] T 92580e541f6e400c94c2cae00a159d83 P 00cb9dff7a9b430c9ad5224aa776edee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:49.115325 7912 raft_consensus.cc:729] T 92580e541f6e400c94c2cae00a159d83 P 00cb9dff7a9b430c9ad5224aa776edee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00cb9dff7a9b430c9ad5224aa776edee, State: Initialized, Role: FOLLOWER
I20260924 03:09:49.115599 7912 consensus_queue.cc:260] T 92580e541f6e400c94c2cae00a159d83 P 00cb9dff7a9b430c9ad5224aa776edee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 } } peers { permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 } }
I20260924 03:09:49.116864 7912 ts_tablet_manager.cc:1428] T 92580e541f6e400c94c2cae00a159d83 P 00cb9dff7a9b430c9ad5224aa776edee: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 03:09:49.117713 7913 tablet_bootstrap.cc:492] T 92580e541f6e400c94c2cae00a159d83 P 765e21fdfb6049d48dcb457debd0e02f: No bootstrap required, opened a new log
I20260924 03:09:49.117883 7913 ts_tablet_manager.cc:1397] T 92580e541f6e400c94c2cae00a159d83 P 765e21fdfb6049d48dcb457debd0e02f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260924 03:09:49.118691 7913 raft_consensus.cc:348] T 92580e541f6e400c94c2cae00a159d83 P 765e21fdfb6049d48dcb457debd0e02f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 } } peers { permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 } }
I20260924 03:09:49.118966 7913 raft_consensus.cc:374] T 92580e541f6e400c94c2cae00a159d83 P 765e21fdfb6049d48dcb457debd0e02f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:49.119076 7913 raft_consensus.cc:729] T 92580e541f6e400c94c2cae00a159d83 P 765e21fdfb6049d48dcb457debd0e02f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 765e21fdfb6049d48dcb457debd0e02f, State: Initialized, Role: FOLLOWER
I20260924 03:09:49.119307 7913 consensus_queue.cc:260] T 92580e541f6e400c94c2cae00a159d83 P 765e21fdfb6049d48dcb457debd0e02f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 } } peers { permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 } }
I20260924 03:09:49.116113 7914 tablet_bootstrap.cc:492] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40: Bootstrap starting.
I20260924 03:09:49.120543 7530 heartbeater.cc:503] Master 127.0.97.61:34973 was elected leader, sending a full tablet report...
I20260924 03:09:49.121899 7914 tablet_bootstrap.cc:654] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:49.123123 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.117362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.1:54757)
0924 03:09:49.122662 (+ 5300us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.122820 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5169,"thread_start_us":353,"threads_started":1}
I20260924 03:09:49.123159 7913 ts_tablet_manager.cc:1428] T 92580e541f6e400c94c2cae00a159d83 P 765e21fdfb6049d48dcb457debd0e02f: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260924 03:09:49.123735 7676 heartbeater.cc:503] Master 127.0.97.61:34973 was elected leader, sending a full tablet report...
I20260924 03:09:49.125308 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.123866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.1:33187)
0924 03:09:49.124987 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.125105 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":979}
I20260924 03:09:49.129959 7914 tablet_bootstrap.cc:492] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40: No bootstrap required, opened a new log
I20260924 03:09:49.130136 7914 ts_tablet_manager.cc:1397] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.003s
I20260924 03:09:49.130793 7914 raft_consensus.cc:348] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 } } peers { permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 } }
I20260924 03:09:49.131002 7914 raft_consensus.cc:374] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:49.131096 7914 raft_consensus.cc:729] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3d53cd8e5c47cbb4da460f4198db40, State: Initialized, Role: FOLLOWER
I20260924 03:09:49.131342 7914 consensus_queue.cc:260] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 } } peers { permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 } }
I20260924 03:09:49.132217 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.121365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.2:59601)
0924 03:09:49.131823 (+ 10458us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.131973 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7840,"thread_start_us":7714,"threads_started":1}
I20260924 03:09:49.132763 7914 ts_tablet_manager.cc:1428] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260924 03:09:49.133842 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.132685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.3:39737)
0924 03:09:49.133500 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.133606 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":510,"threads_started":1}
I20260924 03:09:49.134135 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.133586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.2:59939)
0924 03:09:49.133803 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.133983 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260924 03:09:49.134284 7817 heartbeater.cc:503] Master 127.0.97.61:34973 was elected leader, sending a full tablet report...
I20260924 03:09:49.135210 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.134841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.3:59839)
0924 03:09:49.134998 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.135082 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 03:09:49.457504 7922 raft_consensus.cc:482] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:49.457710 7922 raft_consensus.cc:504] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 } } peers { permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 } }
I20260924 03:09:49.458428 7922 leader_election.cc:288] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 765e21fdfb6049d48dcb457debd0e02f (127.0.97.2:43547), 00cb9dff7a9b430c9ad5224aa776edee (127.0.97.1:37927)
I20260924 03:09:49.459012 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92580e541f6e400c94c2cae00a159d83" candidate_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00cb9dff7a9b430c9ad5224aa776edee" is_pre_election: true
I20260924 03:09:49.459076 7628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92580e541f6e400c94c2cae00a159d83" candidate_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "765e21fdfb6049d48dcb457debd0e02f" is_pre_election: true
I20260924 03:09:49.459286 7483 raft_consensus.cc:2454] T 92580e541f6e400c94c2cae00a159d83 P 00cb9dff7a9b430c9ad5224aa776edee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a3d53cd8e5c47cbb4da460f4198db40 in term 0.
I20260924 03:09:49.459359 7628 raft_consensus.cc:2454] T 92580e541f6e400c94c2cae00a159d83 P 765e21fdfb6049d48dcb457debd0e02f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a3d53cd8e5c47cbb4da460f4198db40 in term 0.
I20260924 03:09:49.459827 7706 leader_election.cc:302] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 765e21fdfb6049d48dcb457debd0e02f, 8a3d53cd8e5c47cbb4da460f4198db40; no voters:
I20260924 03:09:49.460115 7922 raft_consensus.cc:2781] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:49.460224 7922 raft_consensus.cc:482] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:49.460376 7922 raft_consensus.cc:3037] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:49.462724 7922 raft_consensus.cc:504] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 } } peers { permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 } }
I20260924 03:09:49.463354 7922 leader_election.cc:288] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 [CANDIDATE]: Term 1 election: Requested vote from peers 765e21fdfb6049d48dcb457debd0e02f (127.0.97.2:43547), 00cb9dff7a9b430c9ad5224aa776edee (127.0.97.1:37927)
I20260924 03:09:49.463837 7628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92580e541f6e400c94c2cae00a159d83" candidate_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "765e21fdfb6049d48dcb457debd0e02f"
I20260924 03:09:49.463909 7483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92580e541f6e400c94c2cae00a159d83" candidate_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00cb9dff7a9b430c9ad5224aa776edee"
I20260924 03:09:49.464071 7628 raft_consensus.cc:3037] T 92580e541f6e400c94c2cae00a159d83 P 765e21fdfb6049d48dcb457debd0e02f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:49.464102 7483 raft_consensus.cc:3037] T 92580e541f6e400c94c2cae00a159d83 P 00cb9dff7a9b430c9ad5224aa776edee [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:49.466418 7628 raft_consensus.cc:2454] T 92580e541f6e400c94c2cae00a159d83 P 765e21fdfb6049d48dcb457debd0e02f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a3d53cd8e5c47cbb4da460f4198db40 in term 1.
I20260924 03:09:49.466907 7706 leader_election.cc:302] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 765e21fdfb6049d48dcb457debd0e02f, 8a3d53cd8e5c47cbb4da460f4198db40; no voters:
I20260924 03:09:49.467084 7483 raft_consensus.cc:2454] T 92580e541f6e400c94c2cae00a159d83 P 00cb9dff7a9b430c9ad5224aa776edee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a3d53cd8e5c47cbb4da460f4198db40 in term 1.
I20260924 03:09:49.467193 7922 raft_consensus.cc:2781] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:49.470427 7922 raft_consensus.cc:686] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 1 LEADER]: Becoming Leader. State: Replica: 8a3d53cd8e5c47cbb4da460f4198db40, State: Running, Role: LEADER
I20260924 03:09:49.471035 7922 consensus_queue.cc:237] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 } } peers { permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 } }
I20260924 03:09:49.471539 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.471014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.3:49279)
0924 03:09:49.471225 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.471348 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 03:09:49.474927 7163 catalog_manager.cc:5495] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a3d53cd8e5c47cbb4da460f4198db40 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d53cd8e5c47cbb4da460f4198db40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 } health_report { overall_health: UNKNOWN } } }
I20260924 03:09:49.556936 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.551783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57856 (local address 127.0.97.3:37999)
0924 03:09:49.552136 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:49.552141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:49.552156 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:09:49.552243 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:49.552251 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:49.552337 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:49.552531 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:49.552923 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.553595 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.554862 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.555251 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.555388 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:49.555788 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:49.556312 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:49.556461 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:49.556595 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 03:09:49.556671 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:09:49.560829 7628 raft_consensus.cc:1261] T 92580e541f6e400c94c2cae00a159d83 P 765e21fdfb6049d48dcb457debd0e02f [term 1 FOLLOWER]: Refusing update from remote peer 8a3d53cd8e5c47cbb4da460f4198db40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:49.561107 7483 raft_consensus.cc:1261] T 92580e541f6e400c94c2cae00a159d83 P 00cb9dff7a9b430c9ad5224aa776edee [term 1 FOLLOWER]: Refusing update from remote peer 8a3d53cd8e5c47cbb4da460f4198db40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:49.561544 7924 consensus_queue.cc:1035] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "765e21fdfb6049d48dcb457debd0e02f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:49.561934 7922 consensus_queue.cc:1035] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00cb9dff7a9b430c9ad5224aa776edee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:49.564291 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.563676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.1:52871)
0924 03:09:49.563935 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.564103 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260924 03:09:49.565480 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.565032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.2:34657)
0924 03:09:49.565184 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.565310 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 03:09:49.574076 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.573323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.3:56351)
0924 03:09:49.573709 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.573825 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275}
I20260924 03:09:49.575145 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.574574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.2:35263)
0924 03:09:49.574841 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.574963 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260924 03:09:49.573865 7928 mvcc.cc:205] Tried to move back new op lower bound from 7332738619634323456 to 7332738619277750272. Current Snapshot: MvccSnapshot[applied={T|T < 7332738619634323456 or (T in {7332738619634323456})}]
I20260924 03:09:49.581959 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.579074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.3:46307)
0924 03:09:49.579327 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.579414 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260924 03:09:49.582190 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.580458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.97.1:43907)
0924 03:09:49.581622 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.581741 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1076}
I20260924 03:09:49.589906 7163 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:49.590202 7163 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:49.596385 7163 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 782bd86c1d32482598fdc6ed60cc391b: Sending DeleteTablet for 3 replicas of tablet 92580e541f6e400c94c2cae00a159d83
I20260924 03:09:49.597440 7750 tablet_service.cc:1514] Processing DeleteTablet for tablet 92580e541f6e400c94c2cae00a159d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:49 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:57846
I20260924 03:09:49.597604 7609 tablet_service.cc:1514] Processing DeleteTablet for tablet 92580e541f6e400c94c2cae00a159d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:49 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:52612
I20260924 03:09:49.597643 7463 tablet_service.cc:1514] Processing DeleteTablet for tablet 92580e541f6e400c94c2cae00a159d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:49 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:41120
I20260924 03:09:49.600075 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 7327
I20260924 03:09:49.607936 7940 tablet_replica.cc:330] T 92580e541f6e400c94c2cae00a159d83 P 00cb9dff7a9b430c9ad5224aa776edee: stopping tablet replica
I20260924 03:09:49.608224 7940 raft_consensus.cc:2229] T 92580e541f6e400c94c2cae00a159d83 P 00cb9dff7a9b430c9ad5224aa776edee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:49.609872 7939 tablet_replica.cc:330] T 92580e541f6e400c94c2cae00a159d83 P 765e21fdfb6049d48dcb457debd0e02f: stopping tablet replica
I20260924 03:09:49.610131 7939 raft_consensus.cc:2229] T 92580e541f6e400c94c2cae00a159d83 P 765e21fdfb6049d48dcb457debd0e02f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:49.610507 7940 raft_consensus.cc:2258] T 92580e541f6e400c94c2cae00a159d83 P 00cb9dff7a9b430c9ad5224aa776edee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:49.610970 7939 raft_consensus.cc:2258] T 92580e541f6e400c94c2cae00a159d83 P 765e21fdfb6049d48dcb457debd0e02f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:49.612473 7939 ts_tablet_manager.cc:1905] T 92580e541f6e400c94c2cae00a159d83 P 765e21fdfb6049d48dcb457debd0e02f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:49.621968 7939 ts_tablet_manager.cc:1918] T 92580e541f6e400c94c2cae00a159d83 P 765e21fdfb6049d48dcb457debd0e02f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:49.622267 7939 log.cc:1196] T 92580e541f6e400c94c2cae00a159d83 P 765e21fdfb6049d48dcb457debd0e02f: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/92580e541f6e400c94c2cae00a159d83
I20260924 03:09:49.622989 7939 ts_tablet_manager.cc:1939] T 92580e541f6e400c94c2cae00a159d83 P 765e21fdfb6049d48dcb457debd0e02f: Deleting consensus metadata
I20260924 03:09:49.623637 7940 ts_tablet_manager.cc:1905] T 92580e541f6e400c94c2cae00a159d83 P 00cb9dff7a9b430c9ad5224aa776edee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:49.624761 7150 catalog_manager.cc:4841] TS 765e21fdfb6049d48dcb457debd0e02f (127.0.97.2:43547): tablet 92580e541f6e400c94c2cae00a159d83 (table test-table [id=0bf5bfe90d2843a3b52abac5266d6215]) successfully deleted
I20260924 03:09:49.631891 7940 ts_tablet_manager.cc:1918] T 92580e541f6e400c94c2cae00a159d83 P 00cb9dff7a9b430c9ad5224aa776edee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:49.632148 7940 log.cc:1196] T 92580e541f6e400c94c2cae00a159d83 P 00cb9dff7a9b430c9ad5224aa776edee: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/92580e541f6e400c94c2cae00a159d83
I20260924 03:09:49.632907 7940 ts_tablet_manager.cc:1939] T 92580e541f6e400c94c2cae00a159d83 P 00cb9dff7a9b430c9ad5224aa776edee: Deleting consensus metadata
I20260924 03:09:49.634244 7149 catalog_manager.cc:4841] TS 00cb9dff7a9b430c9ad5224aa776edee (127.0.97.1:37927): tablet 92580e541f6e400c94c2cae00a159d83 (table test-table [id=0bf5bfe90d2843a3b52abac5266d6215]) successfully deleted
I20260924 03:09:49.634732 7938 tablet_replica.cc:330] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40: stopping tablet replica
I20260924 03:09:49.634948 7938 raft_consensus.cc:2229] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:09:49.635433 7938 raft_consensus.cc:2258] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:49.636919 7938 ts_tablet_manager.cc:1905] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:49.644083 7938 ts_tablet_manager.cc:1918] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:49.644317 7938 log.cc:1196] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/92580e541f6e400c94c2cae00a159d83
I20260924 03:09:49.644865 7938 ts_tablet_manager.cc:1939] T 92580e541f6e400c94c2cae00a159d83 P 8a3d53cd8e5c47cbb4da460f4198db40: Deleting consensus metadata
I20260924 03:09:49.649058 7147 catalog_manager.cc:4841] TS 8a3d53cd8e5c47cbb4da460f4198db40 (127.0.97.3:37999): tablet 92580e541f6e400c94c2cae00a159d83 (table test-table [id=0bf5bfe90d2843a3b52abac5266d6215]) successfully deleted
I20260924 03:09:49.649313 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.636514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.0.1:34066)
0924 03:09:49.648955 (+ 12441us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.649085 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12280,"thread_start_us":12157,"threads_started":1}
I20260924 03:09:49.650974 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.650522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34603 (local address 127.0.0.1:34070)
0924 03:09:49.650736 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:49.650827 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260924 03:09:49.651782 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.603941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41128 (local address 127.0.97.1:37927)
0924 03:09:49.629814 (+ 25873us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:49.629824 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:49.629853 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:09:49.629936 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:49.629944 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:49.630067 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:49.630215 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:49.632712 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.634186 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.637273 (+ 3087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.638173 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.638320 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:49.643063 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:49.643090 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:49.643118 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:49.643183 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:49.648416 (+ 5233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:49.649504 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:49.649514 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:49.649519 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:49.649722 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:49.650153 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:49.650157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:49.650161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:49.650571 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:49.650931 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:49.651210 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 03:09:49.651425 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25678,"spinlock_wait_cycles":13568,"thread_start_us":25657,"threads_started":1}
I20260924 03:09:49.652593 7523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:49.774402 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 7327
I20260924 03:09:49.797519 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 7540
I20260924 03:09:49.815057 7946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.798555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52624 (local address 127.0.97.2:43547)
0924 03:09:49.799223 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:49.799228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:49.799243 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:09:49.799295 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:49.799298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:49.799368 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:49.799464 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:49.801446 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.802251 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.804115 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.804767 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.804919 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:49.807843 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:49.807864 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:49.807879 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:49.807911 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:49.811880 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:49.812776 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:49.812784 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:49.812787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:49.812979 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:49.813458 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:49.813463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:49.813467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:49.813863 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:49.814152 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:49.814464 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 03:09:49.814714 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":346,"threads_started":1}
I20260924 03:09:49.815959 7669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:49.910241 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 7540
I20260924 03:09:49.927887 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 7687
I20260924 03:09:49.942087 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:49.928889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57872 (local address 127.0.97.3:37999)
0924 03:09:49.929090 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:49.929099 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:49.929126 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:09:49.929185 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:49.929189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:49.929276 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:49.929392 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:49.930705 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.931481 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.932781 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:49.933212 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:49.933349 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:49.935928 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:49.935941 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:49.935959 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:49.935990 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:49.939209 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:49.939947 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:49.939958 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:49.939970 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:49.940118 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:49.940522 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:49.940527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:49.940530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:49.940885 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:49.941227 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:49.941516 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 03:09:49.941753 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 03:09:49.942858 7810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:50.073200 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 7687
I20260924 03:09:50.093629 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 7026
I20260924 03:09:50.094774 7193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:50.222687 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 7026
I20260924 03:09:50.243643 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 7127
I20260924 03:09:50.244774 7380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:50.360903 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 7127
2026-09-24T03:09:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10383 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:09:50 dist-test-slave-5l98 krb5kdc[7954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:09:50 dist-test-slave-5l98 krb5kdc[7954](info): set up 2 sockets
Sep 24 03:09:50 dist-test-slave-5l98 krb5kdc[7954](info): commencing operation
krb5kdc: starting...
W20260924 03:09:51.485668 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:09:51 dist-test-slave-5l98 krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:09:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260924 03:09:51.720897 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45351
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:52.056805 7967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:52.057144 7967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:52.057310 7967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:52.064777 7967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:52.064898 7967 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 03:09:52.064939 7967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:52.065063 7967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:52.065132 7967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:52.065259 7967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:09:52.075680 7967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45351
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:52.077191 7967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:52.078533 7967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:52.090631 7975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:52.090730 7972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:52.090706 7973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:52.091632 7967 server_base.cc:1034] running on GCE node
I20260924 03:09:52.092622 7967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:52.094067 7967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:52.095335 7967 hybrid_clock.cc:648] HybridClock initialized: now 1790219392095275 us; error 41 us; skew 500 ppm
I20260924 03:09:52.095728 7967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:52 dist-test-slave-5l98 krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219392, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:52.103780 7967 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260924 03:09:52.105741 7967 webserver.cc:460] Webserver started at http://127.0.97.62:46053/ using document root <none> and password file <none>
I20260924 03:09:52.106513 7967 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:52.106653 7967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:52.107004 7967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:52.110219 7967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "1bb6f4bf97fe4d9c8b3ba9e986933107"
format_stamp: "Formatted at 2026-09-24 03:09:52 on dist-test-slave-5l98"
server_key: "ca0fcffe9c57dc350a637287647c6622"
server_key_iv: "db24887b975e7532a0dacc090c4824b7"
server_key_version: "encryptionkey@0"
I20260924 03:09:52.111073 7967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "1bb6f4bf97fe4d9c8b3ba9e986933107"
format_stamp: "Formatted at 2026-09-24 03:09:52 on dist-test-slave-5l98"
server_key: "ca0fcffe9c57dc350a637287647c6622"
server_key_iv: "db24887b975e7532a0dacc090c4824b7"
server_key_version: "encryptionkey@0"
I20260924 03:09:52.116724 7967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:09:52.120687 7982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:52.122289 7967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:09:52.122493 7967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "1bb6f4bf97fe4d9c8b3ba9e986933107"
format_stamp: "Formatted at 2026-09-24 03:09:52 on dist-test-slave-5l98"
server_key: "ca0fcffe9c57dc350a637287647c6622"
server_key_iv: "db24887b975e7532a0dacc090c4824b7"
server_key_version: "encryptionkey@0"
I20260924 03:09:52.122726 7967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:52.140175 7967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:52.145063 7967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:52.145454 7967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:52.168838 7967 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45351
I20260924 03:09:52.168872 8043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45351 every 8 connection(s)
I20260924 03:09:52.170374 7967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:09:52.173478 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 7967
I20260924 03:09:52.173702 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:09:52.174479 388 external_mini_cluster.cc:1428] Setting key e025e5d4b67df61f204958ad4e564c08
I20260924 03:09:52.174731 8044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 03:09:52 dist-test-slave-5l98 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 03:09:52.190902 8044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107: Bootstrap starting.
I20260924 03:09:52.195506 8044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:52.197147 8044 log.cc:824] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:52.200881 8044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107: No bootstrap required, opened a new log
I20260924 03:09:52.202071 8047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:52.176556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59228 (local address 127.0.97.62:45351)
0924 03:09:52.177428 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:52.177442 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:52.177485 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:09:52.179646 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:52.179672 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:52.180072 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:52.180408 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:52.182340 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:52.183740 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:52.185542 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:52.186576 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:52.186789 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:52.192187 (+ 5398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:52.192196 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:52.192211 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:52.192253 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:52.197002 (+ 4749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:52.198081 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:52.198089 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:52.198099 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:52.198275 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:52.198720 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:52.198726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:52.198729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:52.199592 (+ 863us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:52.199914 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:52.200433 (+ 519us) server_negotiation.cc:299] Negotiation successful
0924 03:09:52.200954 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":374,"threads_started":1}
I20260924 03:09:52.209476 8044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb6f4bf97fe4d9c8b3ba9e986933107" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45351 } }
I20260924 03:09:52.209908 8044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:52.210026 8044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb6f4bf97fe4d9c8b3ba9e986933107, State: Initialized, Role: FOLLOWER
I20260924 03:09:52.210669 8044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb6f4bf97fe4d9c8b3ba9e986933107" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45351 } }
I20260924 03:09:52.210911 8044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:09:52.211054 8044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:09:52.211230 8044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:52.213780 8044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb6f4bf97fe4d9c8b3ba9e986933107" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45351 } }
I20260924 03:09:52.214310 8044 leader_election.cc:302] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bb6f4bf97fe4d9c8b3ba9e986933107; no voters:
I20260924 03:09:52.214926 8044 leader_election.cc:288] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:09:52.215109 8049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:52.216457 8049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107 [term 1 LEADER]: Becoming Leader. State: Replica: 1bb6f4bf97fe4d9c8b3ba9e986933107, State: Running, Role: LEADER
I20260924 03:09:52.217171 8044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:09:52.217022 8049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb6f4bf97fe4d9c8b3ba9e986933107" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45351 } }
I20260924 03:09:52.219863 8051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bb6f4bf97fe4d9c8b3ba9e986933107. Latest consensus state: current_term: 1 leader_uuid: "1bb6f4bf97fe4d9c8b3ba9e986933107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb6f4bf97fe4d9c8b3ba9e986933107" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45351 } } }
I20260924 03:09:52.220230 8050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bb6f4bf97fe4d9c8b3ba9e986933107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb6f4bf97fe4d9c8b3ba9e986933107" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45351 } } }
I20260924 03:09:52.220577 8050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:52.220579 8051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:52.222357 8058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:09:52.228024 8058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:09:52.236976 8058 catalog_manager.cc:1347] Generated new cluster ID: 0d5682454d524c2a96b2a07e23993ae5
I20260924 03:09:52.237108 8058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:09:52.259227 8058 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:09:52.261160 8058 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:09:52.270859 8058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107: Generated new TSK 0
I20260924 03:09:52.271968 8058 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260924 03:09:52.349763 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45351
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:52.721268 8070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:52.721565 8070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:52.721722 8070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:52.729954 8070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:52.730082 8070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:52.730223 8070 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:09:52.730315 8070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:09:52.741492 8070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:52.742852 8070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:52.744194 8070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:52.761262 8075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:52.761328 8076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:52.761350 8078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:52.762589 8070 server_base.cc:1034] running on GCE node
I20260924 03:09:52.763326 8070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:52.764364 8070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:52.765676 8070 hybrid_clock.cc:648] HybridClock initialized: now 1790219392765580 us; error 72 us; skew 500 ppm
I20260924 03:09:52.766093 8070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:52 dist-test-slave-5l98 krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219392, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:52.769658 8070 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260924 03:09:52.771314 8070 webserver.cc:460] Webserver started at http://127.0.97.1:45207/ using document root <none> and password file <none>
I20260924 03:09:52.772074 8070 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:52.772174 8070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:52.772552 8070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:52.775509 8070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "86caa800b6b04ddca55a6d9e558d8f14"
format_stamp: "Formatted at 2026-09-24 03:09:52 on dist-test-slave-5l98"
server_key: "b5b118b1e6934b87a8d92f75dbf70988"
server_key_iv: "4fac3be9eee0df2835eec59c7a21e187"
server_key_version: "encryptionkey@0"
I20260924 03:09:52.776350 8070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "86caa800b6b04ddca55a6d9e558d8f14"
format_stamp: "Formatted at 2026-09-24 03:09:52 on dist-test-slave-5l98"
server_key: "b5b118b1e6934b87a8d92f75dbf70988"
server_key_iv: "4fac3be9eee0df2835eec59c7a21e187"
server_key_version: "encryptionkey@0"
I20260924 03:09:52.781759 8070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 03:09:52.785692 8085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:52.787451 8070 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 03:09:52.787678 8070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "86caa800b6b04ddca55a6d9e558d8f14"
format_stamp: "Formatted at 2026-09-24 03:09:52 on dist-test-slave-5l98"
server_key: "b5b118b1e6934b87a8d92f75dbf70988"
server_key_iv: "4fac3be9eee0df2835eec59c7a21e187"
server_key_version: "encryptionkey@0"
I20260924 03:09:52.787925 8070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:52.805194 8070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:52.809947 8070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:52.810249 8070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:52.813131 8070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:52.813247 8070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:52.813402 8070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:52.813485 8070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:09:52 dist-test-slave-5l98 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219392, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 03:09:52.841514 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:52.813973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35015 (local address 127.0.97.62:45351)
0924 03:09:52.814520 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:52.814526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:52.815605 (+ 1079us) server_negotiation.cc:407] Connection header received
0924 03:09:52.817928 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:52.817933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:52.818037 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:52.818159 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:52.821035 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:52.822079 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:52.825387 (+ 3308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:52.826152 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:52.826323 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:52.831784 (+ 5461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:52.831792 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:52.831797 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:52.831837 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:52.835503 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:52.836802 (+ 1299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:52.836812 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:52.836818 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:52.837134 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:52.837726 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:52.837731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:52.837735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:52.838022 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:52.838211 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:52.840732 (+ 2521us) server_negotiation.cc:299] Negotiation successful
0924 03:09:52.841023 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":297,"threads_started":1}
I20260924 03:09:52.842319 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:52.814527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45351 (local address 127.0.97.1:35015)
0924 03:09:52.815405 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:52.815452 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:52.817638 (+ 2186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:52.818573 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:52.818589 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:52.819694 (+ 1105us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:52.820757 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:52.820789 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:52.822296 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:52.822304 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:52.825167 (+ 2863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:52.825183 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:52.826439 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:52.826446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:52.826777 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:52.827634 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:52.827685 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:52.831446 (+ 3761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:52.835723 (+ 4277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:52.835734 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:52.835748 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:52.836614 (+ 866us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:52.837298 (+ 684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:52.837302 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:52.837304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:52.837526 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:52.838199 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:52.838215 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:52.838850 (+ 635us) client_negotiation.cc:769] Sending connection context
0924 03:09:52.839431 (+ 581us) client_negotiation.cc:240] Negotiation successful
0924 03:09:52.840159 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":415,"threads_started":1}
I20260924 03:09:52.860303 8070 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45267
I20260924 03:09:52.860412 8205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45267 every 8 connection(s)
I20260924 03:09:52.861801 8070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:09:52.864243 8206 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45351
I20260924 03:09:52.864521 8206 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:52.865350 8206 heartbeater.cc:511] Master 127.0.97.62:45351 requested a full tablet report, sending...
I20260924 03:09:52.865670 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:52.850339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45205 (local address 127.0.97.62:45351)
0924 03:09:52.850541 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:52.850550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:52.851271 (+ 721us) server_negotiation.cc:407] Connection header received
0924 03:09:52.851496 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:52.851503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:52.851603 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:52.851722 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:52.853079 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:52.854107 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:52.855457 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:52.855977 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:52.856144 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:52.858684 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:52.858694 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:52.858701 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:52.858747 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:52.862251 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:52.863256 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:52.863261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:52.863265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:52.863440 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:52.863896 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:52.863901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:52.863904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:52.864357 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:52.864607 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:52.865076 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 03:09:52.865307 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 03:09:52.865702 8188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:52.850328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45351 (local address 127.0.97.1:45205)
0924 03:09:52.851125 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:52.851151 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:52.851349 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:52.851749 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:52.851753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:52.852319 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:52.852931 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:52.852942 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:52.854249 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:52.854254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:52.855294 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:52.855305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:52.856187 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:52.856193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:52.856363 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:52.857156 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:52.857187 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:52.858384 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:52.862428 (+ 4044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:52.862433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:52.862441 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:52.863135 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:52.863589 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:52.863593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:52.863596 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:52.863770 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:52.864604 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:52.864610 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:52.864799 (+ 189us) client_negotiation.cc:769] Sending connection context
0924 03:09:52.864941 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 03:09:52.865240 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":366,"threads_started":1}
I20260924 03:09:52.867223 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 8070
I20260924 03:09:52.867429 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:09:52.868047 7998 ts_manager.cc:194] Registered new tserver with Master: 86caa800b6b04ddca55a6d9e558d8f14 (127.0.97.1:45267)
I20260924 03:09:52.868206 388 external_mini_cluster.cc:1428] Setting key 9f9b329bccb961ad82f3055ff1dd23a2
I20260924 03:09:52.870154 7998 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35015
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260924 03:09:52.921756 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45351
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:53.262921 8210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:53.263208 8210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:53.263371 8210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:53.270856 8210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:53.271005 8210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:53.271155 8210 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:09:53.271258 8210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:09:53.281226 8210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:53.282490 8210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:53.283784 8210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:53.297336 8215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:53.297482 8218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:53.297539 8216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:53.298424 8210 server_base.cc:1034] running on GCE node
I20260924 03:09:53.299034 8210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:53.301224 8210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:53.302500 8210 hybrid_clock.cc:648] HybridClock initialized: now 1790219393302419 us; error 58 us; skew 500 ppm
I20260924 03:09:53.302911 8210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:53 dist-test-slave-5l98 krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219393, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:53.306603 8210 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260924 03:09:53.308377 8210 webserver.cc:460] Webserver started at http://127.0.97.2:37493/ using document root <none> and password file <none>
I20260924 03:09:53.309231 8210 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:53.309321 8210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:53.309667 8210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:53.313277 8210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "e5e206871bfb44ee82ac6f04a7842f0e"
format_stamp: "Formatted at 2026-09-24 03:09:53 on dist-test-slave-5l98"
server_key: "da7735ac8ccdd88e8798df71bf6217b1"
server_key_iv: "913bc29e71a2ebc9121f722ab7884099"
server_key_version: "encryptionkey@0"
I20260924 03:09:53.314105 8210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e5e206871bfb44ee82ac6f04a7842f0e"
format_stamp: "Formatted at 2026-09-24 03:09:53 on dist-test-slave-5l98"
server_key: "da7735ac8ccdd88e8798df71bf6217b1"
server_key_iv: "913bc29e71a2ebc9121f722ab7884099"
server_key_version: "encryptionkey@0"
I20260924 03:09:53.319643 8210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 03:09:53.323683 8225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:53.325299 8210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:09:53.325502 8210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "e5e206871bfb44ee82ac6f04a7842f0e"
format_stamp: "Formatted at 2026-09-24 03:09:53 on dist-test-slave-5l98"
server_key: "da7735ac8ccdd88e8798df71bf6217b1"
server_key_iv: "913bc29e71a2ebc9121f722ab7884099"
server_key_version: "encryptionkey@0"
I20260924 03:09:53.325752 8210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:53.343247 8210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:53.350500 8210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:53.350775 8210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:53.353946 8210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:53.354084 8210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:53.354221 8210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:53.354297 8210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:09:53 dist-test-slave-5l98 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219393, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 03:09:53.381731 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:53.354772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41489 (local address 127.0.97.62:45351)
0924 03:09:53.354953 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:53.354957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:53.356640 (+ 1683us) server_negotiation.cc:407] Connection header received
0924 03:09:53.359083 (+ 2443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:53.359091 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:53.359193 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:53.359267 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:53.362008 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:53.363104 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:53.365806 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:53.366642 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:53.366796 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:53.372872 (+ 6076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:53.372909 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:53.372917 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:53.373007 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:53.376851 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:53.378175 (+ 1324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:53.378181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:53.378185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:53.378340 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:53.379241 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:53.379247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:53.379251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:53.379722 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:53.379882 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:53.381280 (+ 1398us) server_negotiation.cc:299] Negotiation successful
0924 03:09:53.381482 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 03:09:53.383981 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:53.355531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45351 (local address 127.0.97.2:41489)
0924 03:09:53.356407 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:53.356452 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:53.358702 (+ 2250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:53.359856 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:53.359875 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:53.360795 (+ 920us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:53.361776 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:53.361807 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:53.363265 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:53.363277 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:53.365584 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:53.365608 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:53.366912 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:53.366917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:53.367120 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:53.367964 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:53.368006 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:53.372590 (+ 4584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:53.377042 (+ 4452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:53.377056 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:53.377087 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:53.378015 (+ 928us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:53.378470 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:53.378478 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:53.378484 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:53.379038 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:53.379985 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:53.380000 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:53.380712 (+ 712us) client_negotiation.cc:769] Sending connection context
0924 03:09:53.381144 (+ 432us) client_negotiation.cc:240] Negotiation successful
0924 03:09:53.381942 (+ 798us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":365,"threads_started":1}
I20260924 03:09:53.409160 8313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:53.390763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45351 (local address 127.0.97.2:50203)
0924 03:09:53.391597 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:53.391624 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:53.391820 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:53.392463 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:53.392471 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:53.393304 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:53.393788 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:53.393805 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:53.395166 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:53.395174 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:53.397405 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:53.397432 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:53.398627 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:53.398640 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:53.398896 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:53.399615 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:53.399632 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:53.400790 (+ 1158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:53.405507 (+ 4717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:53.405513 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:53.405522 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:53.406272 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:53.406757 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:53.406761 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:53.406764 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:53.406913 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:53.407835 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:53.407842 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:53.408201 (+ 359us) client_negotiation.cc:769] Sending connection context
0924 03:09:53.408436 (+ 235us) client_negotiation.cc:240] Negotiation successful
0924 03:09:53.408779 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":439,"threads_started":1}
I20260924 03:09:53.409543 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:53.390857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50203 (local address 127.0.97.62:45351)
0924 03:09:53.391124 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:53.391135 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:53.391739 (+ 604us) server_negotiation.cc:407] Connection header received
0924 03:09:53.392156 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:53.392162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:53.392234 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:53.392377 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:53.393978 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:53.394874 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:53.397602 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:53.398173 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:53.398323 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:53.401061 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:53.401068 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:53.401073 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:53.401114 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:53.405287 (+ 4173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:53.406471 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:53.406478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:53.406489 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:53.406627 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:53.407118 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:53.407126 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:53.407132 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:53.407639 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:53.407854 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:53.408708 (+ 854us) server_negotiation.cc:299] Negotiation successful
0924 03:09:53.409052 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:09:53.409806 8210 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40267
I20260924 03:09:53.409853 8344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40267 every 8 connection(s)
I20260924 03:09:53.411087 8210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:09:53.413481 8345 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45351
I20260924 03:09:53.413899 8345 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:53.414565 8345 heartbeater.cc:511] Master 127.0.97.62:45351 requested a full tablet report, sending...
I20260924 03:09:53.415122 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 8210
I20260924 03:09:53.415293 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:09:53.415829 388 external_mini_cluster.cc:1428] Setting key f05d1f86a6e7f2a4adb2f55b95483d9b
I20260924 03:09:53.416142 7998 ts_manager.cc:194] Registered new tserver with Master: e5e206871bfb44ee82ac6f04a7842f0e (127.0.97.2:40267)
I20260924 03:09:53.417488 7998 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41489
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260924 03:09:53.469162 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45351
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:53.810153 8349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:53.810521 8349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:53.810693 8349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:53.818099 8349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:53.818281 8349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:53.818445 8349 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:09:53.818564 8349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:09:53.829157 8349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:53.830631 8349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:53.832073 8349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:53.846585 8354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:53.846601 8355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:53.846627 8357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:53.849064 8349 server_base.cc:1034] running on GCE node
I20260924 03:09:53.849862 8349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:53.851052 8349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:53.852337 8349 hybrid_clock.cc:648] HybridClock initialized: now 1790219393852284 us; error 39 us; skew 500 ppm
I20260924 03:09:53.852775 8349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:09:53 dist-test-slave-5l98 krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219393, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:53.856367 8349 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260924 03:09:53.858016 8349 webserver.cc:460] Webserver started at http://127.0.97.3:33323/ using document root <none> and password file <none>
I20260924 03:09:53.858850 8349 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:53.858987 8349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:53.859355 8349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:53.863935 8349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "6173798a8be148beaf8a858af433c383"
format_stamp: "Formatted at 2026-09-24 03:09:53 on dist-test-slave-5l98"
server_key: "a2bfb3785f8a68f40c6b40f1af8b5e0c"
server_key_iv: "c6f1ffbdac89dcd8ae1480d6fa603ce4"
server_key_version: "encryptionkey@0"
I20260924 03:09:53.865090 8349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6173798a8be148beaf8a858af433c383"
format_stamp: "Formatted at 2026-09-24 03:09:53 on dist-test-slave-5l98"
server_key: "a2bfb3785f8a68f40c6b40f1af8b5e0c"
server_key_iv: "c6f1ffbdac89dcd8ae1480d6fa603ce4"
server_key_version: "encryptionkey@0"
I20260924 03:09:53.870975 8349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:09:53.873802 8206 heartbeater.cc:503] Master 127.0.97.62:45351 was elected leader, sending a full tablet report...
I20260924 03:09:53.874888 8364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:53.876402 8349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:09:53.876629 8349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "6173798a8be148beaf8a858af433c383"
format_stamp: "Formatted at 2026-09-24 03:09:53 on dist-test-slave-5l98"
server_key: "a2bfb3785f8a68f40c6b40f1af8b5e0c"
server_key_iv: "c6f1ffbdac89dcd8ae1480d6fa603ce4"
server_key_version: "encryptionkey@0"
I20260924 03:09:53.876829 8349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:53.894665 8349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:53.899331 8349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:53.899581 8349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:53.903080 8349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:53.903205 8349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:53.903360 8349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:53.903447 8349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:09:53 dist-test-slave-5l98 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219393, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 03:09:53.934051 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:53.903665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55855 (local address 127.0.97.62:45351)
0924 03:09:53.903853 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:53.903861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:53.905527 (+ 1666us) server_negotiation.cc:407] Connection header received
0924 03:09:53.909211 (+ 3684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:53.909220 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:53.909338 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:53.909434 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:53.912785 (+ 3351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:53.913891 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:53.917306 (+ 3415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:53.918137 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:53.918324 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:53.924323 (+ 5999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:53.924330 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:53.924357 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:53.924406 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:53.930393 (+ 5987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:53.931438 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:53.931442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:53.931445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:53.931585 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:53.932064 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:53.932070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:53.932073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:53.932372 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:53.932564 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:53.933429 (+ 865us) server_negotiation.cc:299] Negotiation successful
0924 03:09:53.933672 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 03:09:53.935559 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:53.904365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45351 (local address 127.0.97.3:55855)
0924 03:09:53.905312 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:53.905361 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:53.908326 (+ 2965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:53.910274 (+ 1948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:53.910287 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:53.911398 (+ 1111us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:53.912418 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:53.912454 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:53.914112 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:53.914117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:53.917051 (+ 2934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:53.917069 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:53.918711 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:53.918722 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:53.918990 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:53.920107 (+ 1117us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:53.920148 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:53.923908 (+ 3760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:53.930627 (+ 6719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:53.930640 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:53.930663 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:53.931210 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:53.931711 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:53.931716 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:53.931719 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:53.931918 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:53.932549 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:53.932557 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:53.932944 (+ 387us) client_negotiation.cc:769] Sending connection context
0924 03:09:53.933287 (+ 343us) client_negotiation.cc:240] Negotiation successful
0924 03:09:53.933849 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":455,"threads_started":1}
I20260924 03:09:53.961730 8349 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38239
I20260924 03:09:53.962705 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:53.943526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45351 (local address 127.0.97.3:44967)
0924 03:09:53.944843 (+ 1317us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:53.944864 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:53.945035 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:53.945684 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:53.945690 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:53.946582 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:53.947179 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:53.947196 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:53.948978 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:53.948989 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:53.951524 (+ 2535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:53.951542 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:53.952545 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:53.952556 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:53.952800 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:53.953677 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:53.953705 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:53.955226 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:53.959543 (+ 4317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:53.959548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:53.959551 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:53.960290 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:53.960821 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:53.960833 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:53.960839 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:53.961037 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:53.961752 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:53.961756 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:53.961974 (+ 218us) client_negotiation.cc:769] Sending connection context
0924 03:09:53.962107 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 03:09:53.962350 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1120,"thread_start_us":654,"threads_started":1}
I20260924 03:09:53.962949 8483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38239 every 8 connection(s)
I20260924 03:09:53.963016 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:53.943875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44967 (local address 127.0.97.62:45351)
0924 03:09:53.944108 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:53.944118 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:53.944980 (+ 862us) server_negotiation.cc:407] Connection header received
0924 03:09:53.945240 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:53.945248 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:53.945358 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:53.945483 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:53.947419 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:53.948712 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:53.951712 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:53.952261 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:53.952436 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:53.955562 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:53.955571 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:53.955579 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:53.955634 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:53.959366 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:53.960470 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:53.960479 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:53.960483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:53.960674 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:53.961227 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:53.961232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:53.961236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:53.961639 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:53.961804 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:53.962252 (+ 448us) server_negotiation.cc:299] Negotiation successful
0924 03:09:53.962489 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 03:09:53.963374 8349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:09:53.965628 8484 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45351
I20260924 03:09:53.965875 8484 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:53.966519 8484 heartbeater.cc:511] Master 127.0.97.62:45351 requested a full tablet report, sending...
I20260924 03:09:53.968047 7998 ts_manager.cc:194] Registered new tserver with Master: 6173798a8be148beaf8a858af433c383 (127.0.97.3:38239)
I20260924 03:09:53.968977 7998 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55855
I20260924 03:09:53.972416 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 8349
I20260924 03:09:53.972647 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:09:53.973155 388 external_mini_cluster.cc:1428] Setting key 8895995275a042de26416adb85a17426
I20260924 03:09:53.976851 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:09:54 dist-test-slave-5l98 krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219394, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:09:54 dist-test-slave-5l98 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:54 dist-test-slave-5l98 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 03:09:54.029966 8094 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:54.017776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59238 (local address 127.0.97.62:45351)
0924 03:09:54.018044 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:54.018051 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:54.018421 (+ 370us) server_negotiation.cc:407] Connection header received
0924 03:09:54.018505 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:54.018509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:54.018574 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:54.018681 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:54.020182 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.021020 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.022606 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.023021 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.023130 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:54.029664 (+ 6534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59238: BlockingRecv error: recv got EOF from 127.0.0.1:59238 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 03:09:54 dist-test-slave-5l98 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219394, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 03:09:54.048996 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:54.033167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59240 (local address 127.0.97.62:45351)
0924 03:09:54.033438 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:54.033446 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:54.033702 (+ 256us) server_negotiation.cc:407] Connection header received
0924 03:09:54.033920 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:54.033924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:54.033999 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:54.034189 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:54.035357 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.036443 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.037882 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.038387 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.038526 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:54.042678 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:54.042684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:54.042688 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:54.042725 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:54.045806 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:54.046764 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:54.046770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:54.046773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:54.046979 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:54.047424 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:54.047428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:54.047430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:54.047748 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:54.048086 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:54.048375 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 03:09:54.048662 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260924 03:09:54.054497 7998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:09:54.061692 7998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:54.089936 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:54.075378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38239 (local address 127.0.0.1:55850)
0924 03:09:54.076400 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:54.076458 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:54.076719 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:54.079207 (+ 2488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:54.079219 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:54.079277 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:54.079843 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:54.079861 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.083224 (+ 3363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.083234 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:54.086801 (+ 3567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:54.086824 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.088884 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.088891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:54.089107 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:54.089172 (+ 65us) client_negotiation.cc:769] Sending connection context
0924 03:09:54.089284 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 03:09:54.089490 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":486,"thread_start_us":917,"threads_started":1}
I20260924 03:09:54.090440 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:54.075586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40267 (local address 127.0.0.1:57662)
0924 03:09:54.078439 (+ 2853us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:54.078460 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:54.078679 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:54.079867 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:54.079876 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:54.079918 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:54.080552 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:54.080571 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.083250 (+ 2679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.083257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:54.087361 (+ 4104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:54.087372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.089522 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.089529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:54.089741 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:54.089780 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 03:09:54.089862 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 03:09:54.090013 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2699,"mutex_wait_us":29,"thread_start_us":2547,"threads_started":1}
I20260924 03:09:54.091109 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:54.075721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55850 (local address 127.0.97.3:38239)
0924 03:09:54.078477 (+ 2756us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:54.078485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:54.078521 (+ 36us) server_negotiation.cc:407] Connection header received
0924 03:09:54.078630 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:54.078641 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:54.078907 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:54.080424 (+ 1517us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:54.080849 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.082996 (+ 2147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.087015 (+ 4019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.088650 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.090222 (+ 1572us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:54.090456 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:54.090541 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 03:09:54.090733 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2584,"thread_start_us":2421,"threads_started":1}
I20260924 03:09:54.091276 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:54.076067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57662 (local address 127.0.97.2:40267)
0924 03:09:54.077974 (+ 1907us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:54.077983 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:54.078559 (+ 576us) server_negotiation.cc:407] Connection header received
0924 03:09:54.078936 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:54.078950 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:54.079262 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:54.079495 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:54.080871 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.083029 (+ 2158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.087531 (+ 4502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.088968 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.089130 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:54.089356 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:54.090716 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0924 03:09:54.090938 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1747,"thread_start_us":1592,"threads_started":1}
I20260924 03:09:54.095024 8414 tablet_service.cc:1467] Processing CreateTablet for tablet 7e27e44e53f7458f923cf5e684ee87e7 (DEFAULT_TABLE table=test-table [id=e2ce65a533e140d0ab7bdfa7601ba52b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:54.095033 8275 tablet_service.cc:1467] Processing CreateTablet for tablet 7e27e44e53f7458f923cf5e684ee87e7 (DEFAULT_TABLE table=test-table [id=e2ce65a533e140d0ab7bdfa7601ba52b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:54.096653 8414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e27e44e53f7458f923cf5e684ee87e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:54.096692 8275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e27e44e53f7458f923cf5e684ee87e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:54.096796 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:54.078840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46930 (local address 127.0.97.1:45267)
0924 03:09:54.086737 (+ 7897us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:54.086750 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:54.087401 (+ 651us) server_negotiation.cc:407] Connection header received
0924 03:09:54.087515 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:54.087526 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:54.087833 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:54.088052 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:54.088891 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.090597 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.093840 (+ 3243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.095456 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.095620 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:54.095850 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:54.096257 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 03:09:54.096438 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7733,"thread_start_us":7598,"threads_started":1}
I20260924 03:09:54.096801 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:54.078561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45267 (local address 127.0.0.1:46930)
0924 03:09:54.087105 (+ 8544us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:54.087125 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:54.087259 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:54.088081 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:54.088086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:54.088117 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:54.088623 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:54.088641 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.090772 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.090777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:54.093630 (+ 2853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:54.093649 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.095679 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.095688 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:54.096020 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:54.096053 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 03:09:54.096153 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 03:09:54.096308 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8416,"thread_start_us":8238,"threads_started":1}
I20260924 03:09:54.101186 8136 tablet_service.cc:1467] Processing CreateTablet for tablet 7e27e44e53f7458f923cf5e684ee87e7 (DEFAULT_TABLE table=test-table [id=e2ce65a533e140d0ab7bdfa7601ba52b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:54.102792 8136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e27e44e53f7458f923cf5e684ee87e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:54.110684 8508 tablet_bootstrap.cc:492] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383: Bootstrap starting.
I20260924 03:09:54.112735 8509 tablet_bootstrap.cc:492] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e: Bootstrap starting.
I20260924 03:09:54.114711 8508 tablet_bootstrap.cc:654] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:54.116081 8508 log.cc:824] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:54.116158 8509 tablet_bootstrap.cc:654] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:54.117455 8509 log.cc:824] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:54.120055 8511 tablet_bootstrap.cc:492] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14: Bootstrap starting.
I20260924 03:09:54.120110 8508 tablet_bootstrap.cc:492] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383: No bootstrap required, opened a new log
I20260924 03:09:54.120572 8508 ts_tablet_manager.cc:1397] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260924 03:09:54.125201 8511 tablet_bootstrap.cc:654] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:54.126698 8511 log.cc:824] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:54.127597 8509 tablet_bootstrap.cc:492] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e: No bootstrap required, opened a new log
I20260924 03:09:54.127910 8509 ts_tablet_manager.cc:1397] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260924 03:09:54.130913 8511 tablet_bootstrap.cc:492] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14: No bootstrap required, opened a new log
I20260924 03:09:54.130776 8508 raft_consensus.cc:348] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } }
I20260924 03:09:54.131279 8508 raft_consensus.cc:374] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:54.131335 8511 ts_tablet_manager.cc:1397] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260924 03:09:54.131398 8508 raft_consensus.cc:729] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6173798a8be148beaf8a858af433c383, State: Initialized, Role: FOLLOWER
I20260924 03:09:54.132105 8508 consensus_queue.cc:260] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } }
I20260924 03:09:54.133291 8484 heartbeater.cc:503] Master 127.0.97.62:45351 was elected leader, sending a full tablet report...
I20260924 03:09:54.134007 8508 ts_tablet_manager.cc:1428] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260924 03:09:54.139230 8509 raft_consensus.cc:348] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } }
I20260924 03:09:54.139711 8509 raft_consensus.cc:374] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:54.139819 8509 raft_consensus.cc:729] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e206871bfb44ee82ac6f04a7842f0e, State: Initialized, Role: FOLLOWER
I20260924 03:09:54.140645 8509 consensus_queue.cc:260] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } }
I20260924 03:09:54.141793 8345 heartbeater.cc:503] Master 127.0.97.62:45351 was elected leader, sending a full tablet report...
I20260924 03:09:54.142686 8509 ts_tablet_manager.cc:1428] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260924 03:09:54.142479 8511 raft_consensus.cc:348] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } }
I20260924 03:09:54.142938 8511 raft_consensus.cc:374] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:54.143038 8511 raft_consensus.cc:729] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86caa800b6b04ddca55a6d9e558d8f14, State: Initialized, Role: FOLLOWER
I20260924 03:09:54.143710 8511 consensus_queue.cc:260] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } }
I20260924 03:09:54.145721 8511 ts_tablet_manager.cc:1428] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260924 03:09:54.163450 8346 tablet.cc:2369] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:09:54.215420 8485 tablet.cc:2369] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:09:54.311126 8514 raft_consensus.cc:482] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:54.311494 8514 raft_consensus.cc:504] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } }
I20260924 03:09:54.313251 8514 leader_election.cc:288] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5e206871bfb44ee82ac6f04a7842f0e (127.0.97.2:40267), 86caa800b6b04ddca55a6d9e558d8f14 (127.0.97.1:45267)
I20260924 03:09:54.321611 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:54.313804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40267 (local address 127.0.97.3:60613)
0924 03:09:54.314028 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:54.314050 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:54.314212 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:54.314827 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:54.314832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:54.314861 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:54.315094 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:54.315108 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.316633 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.316642 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:54.318870 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:54.318887 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.320835 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.320844 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:54.321122 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:54.321151 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 03:09:54.321247 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 03:09:54.321355 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":1}
I20260924 03:09:54.321764 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:54.314044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60613 (local address 127.0.97.2:40267)
0924 03:09:54.314310 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:54.314319 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:54.314345 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:09:54.314427 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:54.314434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:54.314546 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:54.314678 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:54.315243 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.316397 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.319142 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.320630 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.320804 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:54.320930 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:54.321315 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 03:09:54.321463 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 03:09:54.322125 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:54.314494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54683 (local address 127.0.97.1:45267)
0924 03:09:54.314728 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:54.314736 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:54.315043 (+ 307us) server_negotiation.cc:407] Connection header received
0924 03:09:54.315223 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:54.315228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:54.315320 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:54.315426 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:54.315946 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.317016 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.319316 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.320866 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.321021 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:54.321095 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:54.321715 (+ 620us) server_negotiation.cc:299] Negotiation successful
0924 03:09:54.321886 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 03:09:54.322453 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:54.314210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45267 (local address 127.0.97.3:54683)
0924 03:09:54.314932 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:54.314948 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:54.315070 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:54.315484 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:54.315487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:54.315506 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:54.315787 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:54.315793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.317206 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.317212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:54.319163 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:54.319172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.321301 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.321308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:54.321509 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:54.321556 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 03:09:54.321665 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 03:09:54.322001 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":25,"thread_start_us":441,"threads_started":1}
I20260924 03:09:54.323153 8295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e27e44e53f7458f923cf5e684ee87e7" candidate_uuid: "6173798a8be148beaf8a858af433c383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" is_pre_election: true
I20260924 03:09:54.323519 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e27e44e53f7458f923cf5e684ee87e7" candidate_uuid: "6173798a8be148beaf8a858af433c383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86caa800b6b04ddca55a6d9e558d8f14" is_pre_election: true
I20260924 03:09:54.323776 8295 raft_consensus.cc:2454] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6173798a8be148beaf8a858af433c383 in term 0.
I20260924 03:09:54.324014 8156 raft_consensus.cc:2454] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6173798a8be148beaf8a858af433c383 in term 0.
I20260924 03:09:54.324602 8368 leader_election.cc:302] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6173798a8be148beaf8a858af433c383, e5e206871bfb44ee82ac6f04a7842f0e; no voters:
I20260924 03:09:54.324993 8514 raft_consensus.cc:2781] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:54.325157 8514 raft_consensus.cc:482] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:54.325292 8514 raft_consensus.cc:3037] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:54.327697 8514 raft_consensus.cc:504] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } }
I20260924 03:09:54.328269 8514 leader_election.cc:288] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 [CANDIDATE]: Term 1 election: Requested vote from peers e5e206871bfb44ee82ac6f04a7842f0e (127.0.97.2:40267), 86caa800b6b04ddca55a6d9e558d8f14 (127.0.97.1:45267)
I20260924 03:09:54.328943 8295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e27e44e53f7458f923cf5e684ee87e7" candidate_uuid: "6173798a8be148beaf8a858af433c383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e206871bfb44ee82ac6f04a7842f0e"
I20260924 03:09:54.329072 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e27e44e53f7458f923cf5e684ee87e7" candidate_uuid: "6173798a8be148beaf8a858af433c383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86caa800b6b04ddca55a6d9e558d8f14"
I20260924 03:09:54.329166 8295 raft_consensus.cc:3037] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:54.329315 8156 raft_consensus.cc:3037] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:54.331758 8156 raft_consensus.cc:2454] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6173798a8be148beaf8a858af433c383 in term 1.
I20260924 03:09:54.331768 8295 raft_consensus.cc:2454] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6173798a8be148beaf8a858af433c383 in term 1.
I20260924 03:09:54.332167 8368 leader_election.cc:302] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6173798a8be148beaf8a858af433c383, 86caa800b6b04ddca55a6d9e558d8f14; no voters:
I20260924 03:09:54.332556 8514 raft_consensus.cc:2781] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:54.333132 8514 raft_consensus.cc:686] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 [term 1 LEADER]: Becoming Leader. State: Replica: 6173798a8be148beaf8a858af433c383, State: Running, Role: LEADER
I20260924 03:09:54.333722 8514 consensus_queue.cc:237] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } }
I20260924 03:09:54.338773 7998 catalog_manager.cc:5495] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6173798a8be148beaf8a858af433c383 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6173798a8be148beaf8a858af433c383" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } health_report { overall_health: UNKNOWN } } }
W20260924 03:09:54.365070 8207 tablet.cc:2369] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:09:54.391235 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:54.384722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55860 (local address 127.0.97.3:38239)
0924 03:09:54.385012 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:54.385017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:54.385038 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:09:54.385144 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:54.385147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:54.385245 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:54.385386 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:54.385810 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.386661 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.388400 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.388948 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.389152 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:54.389809 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:54.390402 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:54.390516 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:54.390739 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 03:09:54.390814 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260924 03:09:54.400553 8156 raft_consensus.cc:1261] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14 [term 1 FOLLOWER]: Refusing update from remote peer 6173798a8be148beaf8a858af433c383: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:54.400553 8295 raft_consensus.cc:1261] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e [term 1 FOLLOWER]: Refusing update from remote peer 6173798a8be148beaf8a858af433c383: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:54.401726 8518 consensus_queue.cc:1035] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:54.402109 8514 consensus_queue.cc:1035] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:54.417865 8522 mvcc.cc:205] Tried to move back new op lower bound from 7332738639453130752 to 7332738639195537408. Current Snapshot: MvccSnapshot[applied={T|T < 7332738639453130752}]
I20260924 03:09:54.445943 7998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:54.446267 7998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:54.450574 7998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107: Sending DeleteTablet for 3 replicas of tablet 7e27e44e53f7458f923cf5e684ee87e7
I20260924 03:09:54.451848 8414 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e27e44e53f7458f923cf5e684ee87e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:54 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55850
I20260924 03:09:54.451922 8136 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e27e44e53f7458f923cf5e684ee87e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:54 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46930
I20260924 03:09:54.452623 8275 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e27e44e53f7458f923cf5e684ee87e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:54 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57662
I20260924 03:09:54.452934 8531 tablet_replica.cc:330] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383: stopping tablet replica
I20260924 03:09:54.453392 8531 raft_consensus.cc:2229] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:09:54.454128 8531 raft_consensus.cc:2258] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:54.455309 7998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:09:54.456012 7998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:54.456373 8531 ts_tablet_manager.cc:1905] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:54.465283 8274 tablet_service.cc:1467] Processing CreateTablet for tablet 011061dd43454b8789a54217270b8707 (DEFAULT_TABLE table=test-table [id=a9ad407e196545fb975282808126b9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:54.465905 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 011061dd43454b8789a54217270b8707. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:54.467067 8531 ts_tablet_manager.cc:1918] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:54.467396 8531 log.cc:1196] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/7e27e44e53f7458f923cf5e684ee87e7
I20260924 03:09:54.467837 8533 tablet_replica.cc:330] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e: stopping tablet replica
I20260924 03:09:54.468299 8531 ts_tablet_manager.cc:1939] T 7e27e44e53f7458f923cf5e684ee87e7 P 6173798a8be148beaf8a858af433c383: Deleting consensus metadata
I20260924 03:09:54.468515 8533 raft_consensus.cc:2229] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:54.469084 8533 raft_consensus.cc:2258] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:54.469343 8135 tablet_service.cc:1467] Processing CreateTablet for tablet 011061dd43454b8789a54217270b8707 (DEFAULT_TABLE table=test-table [id=a9ad407e196545fb975282808126b9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:54.469880 8135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 011061dd43454b8789a54217270b8707. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:54.471205 8533 ts_tablet_manager.cc:1905] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:54.472915 7985 catalog_manager.cc:4841] TS 6173798a8be148beaf8a858af433c383 (127.0.97.3:38239): tablet 7e27e44e53f7458f923cf5e684ee87e7 (table test-table [id=e2ce65a533e140d0ab7bdfa7601ba52b]) successfully deleted
I20260924 03:09:54.473335 8414 tablet_service.cc:1467] Processing CreateTablet for tablet 011061dd43454b8789a54217270b8707 (DEFAULT_TABLE table=test-table [id=a9ad407e196545fb975282808126b9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:54.473749 8414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 011061dd43454b8789a54217270b8707. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:54.477329 8511 tablet_bootstrap.cc:492] T 011061dd43454b8789a54217270b8707 P 86caa800b6b04ddca55a6d9e558d8f14: Bootstrap starting.
I20260924 03:09:54.478955 8508 tablet_bootstrap.cc:492] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383: Bootstrap starting.
I20260924 03:09:54.478972 8511 tablet_bootstrap.cc:654] T 011061dd43454b8789a54217270b8707 P 86caa800b6b04ddca55a6d9e558d8f14: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:54.480899 8508 tablet_bootstrap.cc:654] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:54.485916 8509 tablet_bootstrap.cc:492] T 011061dd43454b8789a54217270b8707 P e5e206871bfb44ee82ac6f04a7842f0e: Bootstrap starting.
I20260924 03:09:54.487802 8509 tablet_bootstrap.cc:654] T 011061dd43454b8789a54217270b8707 P e5e206871bfb44ee82ac6f04a7842f0e: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:54.491621 8511 tablet_bootstrap.cc:492] T 011061dd43454b8789a54217270b8707 P 86caa800b6b04ddca55a6d9e558d8f14: No bootstrap required, opened a new log
I20260924 03:09:54.491899 8511 ts_tablet_manager.cc:1397] T 011061dd43454b8789a54217270b8707 P 86caa800b6b04ddca55a6d9e558d8f14: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.008s
I20260924 03:09:54.493206 8511 raft_consensus.cc:348] T 011061dd43454b8789a54217270b8707 P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:54.493587 8511 raft_consensus.cc:374] T 011061dd43454b8789a54217270b8707 P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:54.493731 8511 raft_consensus.cc:729] T 011061dd43454b8789a54217270b8707 P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86caa800b6b04ddca55a6d9e558d8f14, State: Initialized, Role: FOLLOWER
I20260924 03:09:54.494055 8511 consensus_queue.cc:260] T 011061dd43454b8789a54217270b8707 P 86caa800b6b04ddca55a6d9e558d8f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:54.494601 8533 ts_tablet_manager.cc:1918] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:54.494769 8533 log.cc:1196] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/7e27e44e53f7458f923cf5e684ee87e7
I20260924 03:09:54.495309 8533 ts_tablet_manager.cc:1939] T 7e27e44e53f7458f923cf5e684ee87e7 P e5e206871bfb44ee82ac6f04a7842f0e: Deleting consensus metadata
I20260924 03:09:54.495913 8511 ts_tablet_manager.cc:1428] T 011061dd43454b8789a54217270b8707 P 86caa800b6b04ddca55a6d9e558d8f14: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 03:09:54.497601 7986 catalog_manager.cc:4841] TS e5e206871bfb44ee82ac6f04a7842f0e (127.0.97.2:40267): tablet 7e27e44e53f7458f923cf5e684ee87e7 (table test-table [id=e2ce65a533e140d0ab7bdfa7601ba52b]) successfully deleted
I20260924 03:09:54.499068 8508 tablet_bootstrap.cc:492] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383: No bootstrap required, opened a new log
I20260924 03:09:54.499334 8508 ts_tablet_manager.cc:1397] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.004s
I20260924 03:09:54.500546 8508 raft_consensus.cc:348] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:54.500941 8508 raft_consensus.cc:374] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:54.501106 8508 raft_consensus.cc:729] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6173798a8be148beaf8a858af433c383, State: Initialized, Role: FOLLOWER
I20260924 03:09:54.501555 8508 consensus_queue.cc:260] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:54.502652 8508 ts_tablet_manager.cc:1428] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 03:09:54.503027 8532 tablet_replica.cc:330] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14: stopping tablet replica
I20260924 03:09:54.503305 8509 tablet_bootstrap.cc:492] T 011061dd43454b8789a54217270b8707 P e5e206871bfb44ee82ac6f04a7842f0e: No bootstrap required, opened a new log
I20260924 03:09:54.503461 8509 ts_tablet_manager.cc:1397] T 011061dd43454b8789a54217270b8707 P e5e206871bfb44ee82ac6f04a7842f0e: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.004s
I20260924 03:09:54.503453 8532 raft_consensus.cc:2229] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:54.503877 8532 raft_consensus.cc:2258] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:54.506119 8532 ts_tablet_manager.cc:1905] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:54.505030 8509 raft_consensus.cc:348] T 011061dd43454b8789a54217270b8707 P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:54.506245 8509 raft_consensus.cc:374] T 011061dd43454b8789a54217270b8707 P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:54.506352 8509 raft_consensus.cc:729] T 011061dd43454b8789a54217270b8707 P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e206871bfb44ee82ac6f04a7842f0e, State: Initialized, Role: FOLLOWER
I20260924 03:09:54.506652 8509 consensus_queue.cc:260] T 011061dd43454b8789a54217270b8707 P e5e206871bfb44ee82ac6f04a7842f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:54.508008 8509 ts_tablet_manager.cc:1428] T 011061dd43454b8789a54217270b8707 P e5e206871bfb44ee82ac6f04a7842f0e: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260924 03:09:54.512650 8532 ts_tablet_manager.cc:1918] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:54.512835 8532 log.cc:1196] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/7e27e44e53f7458f923cf5e684ee87e7
I20260924 03:09:54.513329 8532 ts_tablet_manager.cc:1939] T 7e27e44e53f7458f923cf5e684ee87e7 P 86caa800b6b04ddca55a6d9e558d8f14: Deleting consensus metadata
I20260924 03:09:54.514889 7986 catalog_manager.cc:4841] TS 86caa800b6b04ddca55a6d9e558d8f14 (127.0.97.1:45267): tablet 7e27e44e53f7458f923cf5e684ee87e7 (table test-table [id=e2ce65a533e140d0ab7bdfa7601ba52b]) successfully deleted
I20260924 03:09:54.798663 8514 raft_consensus.cc:482] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:54.799021 8514 raft_consensus.cc:504] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:54.799839 8514 leader_election.cc:288] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86caa800b6b04ddca55a6d9e558d8f14 (127.0.97.1:45267), e5e206871bfb44ee82ac6f04a7842f0e (127.0.97.2:40267)
I20260924 03:09:54.800631 8295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "011061dd43454b8789a54217270b8707" candidate_uuid: "6173798a8be148beaf8a858af433c383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" is_pre_election: true
I20260924 03:09:54.800666 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "011061dd43454b8789a54217270b8707" candidate_uuid: "6173798a8be148beaf8a858af433c383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86caa800b6b04ddca55a6d9e558d8f14" is_pre_election: true
I20260924 03:09:54.800907 8295 raft_consensus.cc:2454] T 011061dd43454b8789a54217270b8707 P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6173798a8be148beaf8a858af433c383 in term 0.
I20260924 03:09:54.800915 8156 raft_consensus.cc:2454] T 011061dd43454b8789a54217270b8707 P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6173798a8be148beaf8a858af433c383 in term 0.
I20260924 03:09:54.801420 8368 leader_election.cc:302] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6173798a8be148beaf8a858af433c383, 86caa800b6b04ddca55a6d9e558d8f14; no voters:
I20260924 03:09:54.801836 8514 raft_consensus.cc:2781] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:54.801985 8514 raft_consensus.cc:482] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:54.802090 8514 raft_consensus.cc:3037] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:54.805716 8514 raft_consensus.cc:504] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:54.806555 8514 leader_election.cc:288] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 [CANDIDATE]: Term 1 election: Requested vote from peers 86caa800b6b04ddca55a6d9e558d8f14 (127.0.97.1:45267), e5e206871bfb44ee82ac6f04a7842f0e (127.0.97.2:40267)
I20260924 03:09:54.807330 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "011061dd43454b8789a54217270b8707" candidate_uuid: "6173798a8be148beaf8a858af433c383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86caa800b6b04ddca55a6d9e558d8f14"
I20260924 03:09:54.807331 8295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "011061dd43454b8789a54217270b8707" candidate_uuid: "6173798a8be148beaf8a858af433c383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e206871bfb44ee82ac6f04a7842f0e"
I20260924 03:09:54.807569 8295 raft_consensus.cc:3037] T 011061dd43454b8789a54217270b8707 P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:54.807561 8156 raft_consensus.cc:3037] T 011061dd43454b8789a54217270b8707 P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:54.810623 8295 raft_consensus.cc:2454] T 011061dd43454b8789a54217270b8707 P e5e206871bfb44ee82ac6f04a7842f0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6173798a8be148beaf8a858af433c383 in term 1.
I20260924 03:09:54.810621 8156 raft_consensus.cc:2454] T 011061dd43454b8789a54217270b8707 P 86caa800b6b04ddca55a6d9e558d8f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6173798a8be148beaf8a858af433c383 in term 1.
I20260924 03:09:54.811112 8368 leader_election.cc:302] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6173798a8be148beaf8a858af433c383, 86caa800b6b04ddca55a6d9e558d8f14; no voters:
I20260924 03:09:54.811502 8514 raft_consensus.cc:2781] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:54.811676 8514 raft_consensus.cc:686] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 [term 1 LEADER]: Becoming Leader. State: Replica: 6173798a8be148beaf8a858af433c383, State: Running, Role: LEADER
I20260924 03:09:54.811961 8514 consensus_queue.cc:237] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:54.815053 7998 catalog_manager.cc:5495] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6173798a8be148beaf8a858af433c383 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6173798a8be148beaf8a858af433c383" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } health_report { overall_health: HEALTHY } } }
Sep 24 03:09:54 dist-test-slave-5l98 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219392, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 03:09:54.907526 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:54.891394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45351 (local address 127.0.0.1:59256)
0924 03:09:54.892309 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:54.892336 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:54.892540 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:54.893011 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:54.893018 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:54.893689 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:54.893976 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:54.893988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.895643 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.895646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:54.896720 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:54.896756 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.897697 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.897702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:54.897835 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:54.898649 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:54.898666 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:54.900966 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:54.904839 (+ 3873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:54.904851 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:54.904856 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:54.905330 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:54.905746 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:54.905749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:54.905751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:54.905890 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:54.906610 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:54.906617 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:54.906753 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 03:09:54.906897 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 03:09:54.907234 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":586,"threads_started":1}
I20260924 03:09:54.907877 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:54.891401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59256 (local address 127.0.97.62:45351)
0924 03:09:54.892696 (+ 1295us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:54.892700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:54.892716 (+ 16us) server_negotiation.cc:407] Connection header received
0924 03:09:54.892753 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:54.892756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:54.892813 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:54.892927 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:54.894165 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.895463 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.896893 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:54.897517 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:54.897639 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:54.901180 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:54.901186 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:54.901190 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:54.901222 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:54.904682 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:54.905478 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:54.905483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:54.905487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:54.905634 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:54.906093 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:54.906098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:54.906101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:54.906440 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:54.906581 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:54.907047 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 03:09:54.907446 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":1171,"thread_start_us":957,"threads_started":1}
I20260924 03:09:54.914367 7998 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59256:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 03:09:54.915462 7998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:54.923363 8414 tablet_service.cc:1467] Processing CreateTablet for tablet 91ff8082161240bb8e56c3e15ef72283 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93c009fcb063474080c5348bc87d638a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:09:54.923331 8274 tablet_service.cc:1467] Processing CreateTablet for tablet 91ff8082161240bb8e56c3e15ef72283 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93c009fcb063474080c5348bc87d638a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:09:54.923934 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ff8082161240bb8e56c3e15ef72283. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:54.923926 8414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ff8082161240bb8e56c3e15ef72283. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:54.923801 8136 tablet_service.cc:1467] Processing CreateTablet for tablet 91ff8082161240bb8e56c3e15ef72283 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93c009fcb063474080c5348bc87d638a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:09:54.924414 8136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ff8082161240bb8e56c3e15ef72283. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:54.930647 8511 tablet_bootstrap.cc:492] T 91ff8082161240bb8e56c3e15ef72283 P 86caa800b6b04ddca55a6d9e558d8f14: Bootstrap starting.
I20260924 03:09:54.931106 8509 tablet_bootstrap.cc:492] T 91ff8082161240bb8e56c3e15ef72283 P e5e206871bfb44ee82ac6f04a7842f0e: Bootstrap starting.
I20260924 03:09:54.931517 8508 tablet_bootstrap.cc:492] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383: Bootstrap starting.
I20260924 03:09:54.932278 8511 tablet_bootstrap.cc:654] T 91ff8082161240bb8e56c3e15ef72283 P 86caa800b6b04ddca55a6d9e558d8f14: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:54.932723 8509 tablet_bootstrap.cc:654] T 91ff8082161240bb8e56c3e15ef72283 P e5e206871bfb44ee82ac6f04a7842f0e: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:54.933244 8508 tablet_bootstrap.cc:654] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:54.934823 8511 tablet_bootstrap.cc:492] T 91ff8082161240bb8e56c3e15ef72283 P 86caa800b6b04ddca55a6d9e558d8f14: No bootstrap required, opened a new log
I20260924 03:09:54.934995 8511 ts_tablet_manager.cc:1397] T 91ff8082161240bb8e56c3e15ef72283 P 86caa800b6b04ddca55a6d9e558d8f14: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260924 03:09:54.935658 8511 raft_consensus.cc:348] T 91ff8082161240bb8e56c3e15ef72283 P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } }
I20260924 03:09:54.935885 8511 raft_consensus.cc:374] T 91ff8082161240bb8e56c3e15ef72283 P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:54.935966 8511 raft_consensus.cc:729] T 91ff8082161240bb8e56c3e15ef72283 P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86caa800b6b04ddca55a6d9e558d8f14, State: Initialized, Role: FOLLOWER
I20260924 03:09:54.936045 8508 tablet_bootstrap.cc:492] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383: No bootstrap required, opened a new log
I20260924 03:09:54.936192 8508 ts_tablet_manager.cc:1397] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 03:09:54.936393 8511 consensus_queue.cc:260] T 91ff8082161240bb8e56c3e15ef72283 P 86caa800b6b04ddca55a6d9e558d8f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } }
I20260924 03:09:54.936860 8516 tablet_replica.cc:439] T 91ff8082161240bb8e56c3e15ef72283 P 86caa800b6b04ddca55a6d9e558d8f14: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } }
I20260924 03:09:54.937160 8516 tablet_replica.cc:442] T 91ff8082161240bb8e56c3e15ef72283 P 86caa800b6b04ddca55a6d9e558d8f14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:54.937668 8511 ts_tablet_manager.cc:1428] T 91ff8082161240bb8e56c3e15ef72283 P 86caa800b6b04ddca55a6d9e558d8f14: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 03:09:54.937916 8509 tablet_bootstrap.cc:492] T 91ff8082161240bb8e56c3e15ef72283 P e5e206871bfb44ee82ac6f04a7842f0e: No bootstrap required, opened a new log
I20260924 03:09:54.937965 8508 raft_consensus.cc:348] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } }
I20260924 03:09:54.938124 8509 ts_tablet_manager.cc:1397] T 91ff8082161240bb8e56c3e15ef72283 P e5e206871bfb44ee82ac6f04a7842f0e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260924 03:09:54.938226 8508 raft_consensus.cc:374] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:54.938323 8508 raft_consensus.cc:729] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6173798a8be148beaf8a858af433c383, State: Initialized, Role: FOLLOWER
I20260924 03:09:54.938561 8508 consensus_queue.cc:260] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } }
I20260924 03:09:54.938906 8509 raft_consensus.cc:348] T 91ff8082161240bb8e56c3e15ef72283 P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } }
I20260924 03:09:54.939126 8508 ts_tablet_manager.cc:1428] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 03:09:54.938947 8514 tablet_replica.cc:439] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } }
I20260924 03:09:54.939196 8509 raft_consensus.cc:374] T 91ff8082161240bb8e56c3e15ef72283 P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:54.939270 8514 tablet_replica.cc:442] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:54.939275 8509 raft_consensus.cc:729] T 91ff8082161240bb8e56c3e15ef72283 P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e206871bfb44ee82ac6f04a7842f0e, State: Initialized, Role: FOLLOWER
I20260924 03:09:54.939512 8509 consensus_queue.cc:260] T 91ff8082161240bb8e56c3e15ef72283 P e5e206871bfb44ee82ac6f04a7842f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } }
I20260924 03:09:54.940073 8509 ts_tablet_manager.cc:1428] T 91ff8082161240bb8e56c3e15ef72283 P e5e206871bfb44ee82ac6f04a7842f0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260924 03:09:54.939998 8515 tablet_replica.cc:439] T 91ff8082161240bb8e56c3e15ef72283 P e5e206871bfb44ee82ac6f04a7842f0e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } }
I20260924 03:09:54.940356 8515 tablet_replica.cc:442] T 91ff8082161240bb8e56c3e15ef72283 P e5e206871bfb44ee82ac6f04a7842f0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:55.212107 8514 raft_consensus.cc:482] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:55.212314 8514 raft_consensus.cc:504] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } }
I20260924 03:09:55.213090 8514 leader_election.cc:288] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5e206871bfb44ee82ac6f04a7842f0e (127.0.97.2:40267), 86caa800b6b04ddca55a6d9e558d8f14 (127.0.97.1:45267)
I20260924 03:09:55.213826 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ff8082161240bb8e56c3e15ef72283" candidate_uuid: "6173798a8be148beaf8a858af433c383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86caa800b6b04ddca55a6d9e558d8f14" is_pre_election: true
I20260924 03:09:55.213824 8295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ff8082161240bb8e56c3e15ef72283" candidate_uuid: "6173798a8be148beaf8a858af433c383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" is_pre_election: true
I20260924 03:09:55.214087 8156 raft_consensus.cc:2454] T 91ff8082161240bb8e56c3e15ef72283 P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6173798a8be148beaf8a858af433c383 in term 0.
I20260924 03:09:55.214107 8295 raft_consensus.cc:2454] T 91ff8082161240bb8e56c3e15ef72283 P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6173798a8be148beaf8a858af433c383 in term 0.
I20260924 03:09:55.214593 8368 leader_election.cc:302] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6173798a8be148beaf8a858af433c383, 86caa800b6b04ddca55a6d9e558d8f14; no voters:
I20260924 03:09:55.214900 8514 raft_consensus.cc:2781] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:55.215008 8514 raft_consensus.cc:482] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:55.215123 8514 raft_consensus.cc:3037] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:55.217304 8514 raft_consensus.cc:504] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } }
I20260924 03:09:55.218065 8514 leader_election.cc:288] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383 [CANDIDATE]: Term 1 election: Requested vote from peers e5e206871bfb44ee82ac6f04a7842f0e (127.0.97.2:40267), 86caa800b6b04ddca55a6d9e558d8f14 (127.0.97.1:45267)
I20260924 03:09:55.218600 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ff8082161240bb8e56c3e15ef72283" candidate_uuid: "6173798a8be148beaf8a858af433c383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86caa800b6b04ddca55a6d9e558d8f14"
I20260924 03:09:55.218602 8295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ff8082161240bb8e56c3e15ef72283" candidate_uuid: "6173798a8be148beaf8a858af433c383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e206871bfb44ee82ac6f04a7842f0e"
I20260924 03:09:55.218843 8156 raft_consensus.cc:3037] T 91ff8082161240bb8e56c3e15ef72283 P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:55.218860 8295 raft_consensus.cc:3037] T 91ff8082161240bb8e56c3e15ef72283 P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:55.221086 8156 raft_consensus.cc:2454] T 91ff8082161240bb8e56c3e15ef72283 P 86caa800b6b04ddca55a6d9e558d8f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6173798a8be148beaf8a858af433c383 in term 1.
I20260924 03:09:55.221424 8295 raft_consensus.cc:2454] T 91ff8082161240bb8e56c3e15ef72283 P e5e206871bfb44ee82ac6f04a7842f0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6173798a8be148beaf8a858af433c383 in term 1.
I20260924 03:09:55.221582 8368 leader_election.cc:302] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6173798a8be148beaf8a858af433c383, 86caa800b6b04ddca55a6d9e558d8f14; no voters:
I20260924 03:09:55.221889 8514 raft_consensus.cc:2781] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:55.222081 8514 raft_consensus.cc:686] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383 [term 1 LEADER]: Becoming Leader. State: Replica: 6173798a8be148beaf8a858af433c383, State: Running, Role: LEADER
I20260924 03:09:55.222343 8514 consensus_queue.cc:237] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } }
I20260924 03:09:55.223938 8528 tablet_replica.cc:439] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383: TxnStatusTablet state changed. Reason: New leader 6173798a8be148beaf8a858af433c383. Latest consensus state: current_term: 1 leader_uuid: "6173798a8be148beaf8a858af433c383" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } }
I20260924 03:09:55.224193 8528 tablet_replica.cc:442] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:09:55.224848 8549 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 03:09:55.225139 8549 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 03:09:55.225836 7997 catalog_manager.cc:5495] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6173798a8be148beaf8a858af433c383 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6173798a8be148beaf8a858af433c383" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } health_report { overall_health: UNKNOWN } } }
I20260924 03:09:55.240847 8528 consensus_queue.cc:1035] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:55.251068 8514 consensus_queue.cc:1035] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:55.315146 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:55.308383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.0.97.3:38239)
0924 03:09:55.309152 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:55.309157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:55.309178 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:09:55.309240 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:55.309243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:55.309308 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:55.309396 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:55.309942 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:55.310913 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:55.312916 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:55.313640 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:55.313770 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:55.314319 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:55.314427 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:55.314507 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:55.314693 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 03:09:55.314758 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":456,"threads_started":1}
I20260924 03:09:55.315148 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:55.308375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38239 (local address 127.0.0.1:55870)
0924 03:09:55.308687 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:55.308709 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:55.308866 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:55.309453 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:55.309458 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:55.309472 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:09:55.309671 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:55.309681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:55.311085 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:55.311091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:55.312725 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:55.312741 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:55.313803 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:55.313810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:55.314013 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:55.314099 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:09:55.314543 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:09:55.314569 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 03:09:55.314660 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 03:09:55.314820 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 03:09:55.319628 8295 raft_consensus.cc:1261] T 91ff8082161240bb8e56c3e15ef72283 P e5e206871bfb44ee82ac6f04a7842f0e [term 1 FOLLOWER]: Refusing update from remote peer 6173798a8be148beaf8a858af433c383: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:55.319640 8156 raft_consensus.cc:1261] T 91ff8082161240bb8e56c3e15ef72283 P 86caa800b6b04ddca55a6d9e558d8f14 [term 1 FOLLOWER]: Refusing update from remote peer 6173798a8be148beaf8a858af433c383: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:55.320299 8554 consensus_queue.cc:1035] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:55.320726 8528 consensus_queue.cc:1035] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:55.323984 8515 tablet_replica.cc:439] T 91ff8082161240bb8e56c3e15ef72283 P e5e206871bfb44ee82ac6f04a7842f0e: TxnStatusTablet state changed. Reason: New leader 6173798a8be148beaf8a858af433c383. Latest consensus state: current_term: 1 leader_uuid: "6173798a8be148beaf8a858af433c383" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } }
I20260924 03:09:55.324244 8515 tablet_replica.cc:442] T 91ff8082161240bb8e56c3e15ef72283 P e5e206871bfb44ee82ac6f04a7842f0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:55.327379 8556 mvcc.cc:205] Tried to move back new op lower bound from 7332738643221512192 to 7332738642832863232. Current Snapshot: MvccSnapshot[applied={T|T < 7332738643221512192}]
I20260924 03:09:55.330693 8514 tablet_replica.cc:439] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6173798a8be148beaf8a858af433c383" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } }
I20260924 03:09:55.331045 8514 tablet_replica.cc:442] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:09:55.332324 8515 tablet_replica.cc:439] T 91ff8082161240bb8e56c3e15ef72283 P e5e206871bfb44ee82ac6f04a7842f0e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6173798a8be148beaf8a858af433c383" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } }
I20260924 03:09:55.332590 8515 tablet_replica.cc:442] T 91ff8082161240bb8e56c3e15ef72283 P e5e206871bfb44ee82ac6f04a7842f0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:55.335736 8516 tablet_replica.cc:439] T 91ff8082161240bb8e56c3e15ef72283 P 86caa800b6b04ddca55a6d9e558d8f14: TxnStatusTablet state changed. Reason: New leader 6173798a8be148beaf8a858af433c383. Latest consensus state: current_term: 1 leader_uuid: "6173798a8be148beaf8a858af433c383" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } }
I20260924 03:09:55.336021 8516 tablet_replica.cc:442] T 91ff8082161240bb8e56c3e15ef72283 P 86caa800b6b04ddca55a6d9e558d8f14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:55.336776 8554 tablet_replica.cc:439] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6173798a8be148beaf8a858af433c383" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } }
I20260924 03:09:55.337082 8554 tablet_replica.cc:442] T 91ff8082161240bb8e56c3e15ef72283 P 6173798a8be148beaf8a858af433c383: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:09:55.340153 8516 tablet_replica.cc:439] T 91ff8082161240bb8e56c3e15ef72283 P 86caa800b6b04ddca55a6d9e558d8f14: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6173798a8be148beaf8a858af433c383" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } } peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } }
I20260924 03:09:55.340407 8516 tablet_replica.cc:442] T 91ff8082161240bb8e56c3e15ef72283 P 86caa800b6b04ddca55a6d9e558d8f14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:55.355058 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:55.349200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49433 (local address 127.0.97.3:38239)
0924 03:09:55.349457 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:55.349462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:55.349992 (+ 530us) server_negotiation.cc:407] Connection header received
0924 03:09:55.350201 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:55.350206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:55.350271 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:55.350388 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:55.350910 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:55.351697 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:55.353264 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:55.353675 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:55.353880 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:55.354273 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:55.354369 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:55.354471 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:55.354617 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 03:09:55.354687 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":60}
I20260924 03:09:55.355144 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:55.349173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38239 (local address 127.0.97.3:49433)
0924 03:09:55.349886 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:55.349903 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:55.350083 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:55.350398 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:55.350402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:55.350412 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:09:55.350752 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:55.350761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:55.351821 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:55.351832 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:55.353105 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:55.353116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:55.353917 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:55.353923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:55.354042 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:55.354093 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:09:55.354472 (+ 379us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:09:55.354497 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 03:09:55.354590 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 03:09:55.354716 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":446,"threads_started":1}
I20260924 03:09:55.431881 7997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:55.432183 7997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:55.435883 7997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107: Sending DeleteTablet for 3 replicas of tablet 011061dd43454b8789a54217270b8707
I20260924 03:09:55.436815 8136 tablet_service.cc:1514] Processing DeleteTablet for tablet 011061dd43454b8789a54217270b8707 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:55 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46930
I20260924 03:09:55.437096 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet 011061dd43454b8789a54217270b8707 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:55 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55850
I20260924 03:09:55.437438 8274 tablet_service.cc:1514] Processing DeleteTablet for tablet 011061dd43454b8789a54217270b8707 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:55 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57662
I20260924 03:09:55.454140 8570 tablet_replica.cc:330] T 011061dd43454b8789a54217270b8707 P e5e206871bfb44ee82ac6f04a7842f0e: stopping tablet replica
I20260924 03:09:55.454452 8570 raft_consensus.cc:2229] T 011061dd43454b8789a54217270b8707 P e5e206871bfb44ee82ac6f04a7842f0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:55.454737 8570 raft_consensus.cc:2258] T 011061dd43454b8789a54217270b8707 P e5e206871bfb44ee82ac6f04a7842f0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:55.455487 8569 tablet_replica.cc:330] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383: stopping tablet replica
I20260924 03:09:55.455722 8569 raft_consensus.cc:2229] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:09:55.456195 8569 raft_consensus.cc:2258] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:55.456576 8570 ts_tablet_manager.cc:1905] T 011061dd43454b8789a54217270b8707 P e5e206871bfb44ee82ac6f04a7842f0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:55.456775 8568 tablet_replica.cc:330] T 011061dd43454b8789a54217270b8707 P 86caa800b6b04ddca55a6d9e558d8f14: stopping tablet replica
I20260924 03:09:55.456969 8568 raft_consensus.cc:2229] T 011061dd43454b8789a54217270b8707 P 86caa800b6b04ddca55a6d9e558d8f14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:55.457190 8568 raft_consensus.cc:2258] T 011061dd43454b8789a54217270b8707 P 86caa800b6b04ddca55a6d9e558d8f14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:55.457620 8569 ts_tablet_manager.cc:1905] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:55.458292 8568 ts_tablet_manager.cc:1905] T 011061dd43454b8789a54217270b8707 P 86caa800b6b04ddca55a6d9e558d8f14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:55.465431 8570 ts_tablet_manager.cc:1918] T 011061dd43454b8789a54217270b8707 P e5e206871bfb44ee82ac6f04a7842f0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:09:55.465431 8568 ts_tablet_manager.cc:1918] T 011061dd43454b8789a54217270b8707 P 86caa800b6b04ddca55a6d9e558d8f14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:09:55.465600 8568 log.cc:1196] T 011061dd43454b8789a54217270b8707 P 86caa800b6b04ddca55a6d9e558d8f14: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/011061dd43454b8789a54217270b8707
I20260924 03:09:55.465596 8570 log.cc:1196] T 011061dd43454b8789a54217270b8707 P e5e206871bfb44ee82ac6f04a7842f0e: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/011061dd43454b8789a54217270b8707
I20260924 03:09:55.466037 8568 ts_tablet_manager.cc:1939] T 011061dd43454b8789a54217270b8707 P 86caa800b6b04ddca55a6d9e558d8f14: Deleting consensus metadata
I20260924 03:09:55.466037 8570 ts_tablet_manager.cc:1939] T 011061dd43454b8789a54217270b8707 P e5e206871bfb44ee82ac6f04a7842f0e: Deleting consensus metadata
I20260924 03:09:55.467245 7986 catalog_manager.cc:4841] TS e5e206871bfb44ee82ac6f04a7842f0e (127.0.97.2:40267): tablet 011061dd43454b8789a54217270b8707 (table test-table [id=a9ad407e196545fb975282808126b9b8]) successfully deleted
I20260924 03:09:55.467516 7986 catalog_manager.cc:4841] TS 86caa800b6b04ddca55a6d9e558d8f14 (127.0.97.1:45267): tablet 011061dd43454b8789a54217270b8707 (table test-table [id=a9ad407e196545fb975282808126b9b8]) successfully deleted
I20260924 03:09:55.471385 8569 ts_tablet_manager.cc:1918] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:09:55.471557 8569 log.cc:1196] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/011061dd43454b8789a54217270b8707
I20260924 03:09:55.472344 8569 ts_tablet_manager.cc:1939] T 011061dd43454b8789a54217270b8707 P 6173798a8be148beaf8a858af433c383: Deleting consensus metadata
I20260924 03:09:55.473614 7985 catalog_manager.cc:4841] TS 6173798a8be148beaf8a858af433c383 (127.0.97.3:38239): tablet 011061dd43454b8789a54217270b8707 (table test-table [id=a9ad407e196545fb975282808126b9b8]) successfully deleted
Sep 24 03:09:55 dist-test-slave-5l98 krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:09:55 dist-test-slave-5l98 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:55 dist-test-slave-5l98 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 03:09:55.514217 8578 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:55.501695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.0.97.62:45351)
0924 03:09:55.502494 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:55.502499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:55.502516 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:09:55.502569 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:55.502572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:55.502630 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:55.502721 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:55.504362 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:55.505431 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:55.507142 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:55.507581 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:55.507687 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:55.513768 (+ 6081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59264: BlockingRecv error: recv got EOF from 127.0.0.1:59264 (error 108)
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":572,"threads_started":1}
Sep 24 03:09:55 dist-test-slave-5l98 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 03:09:55.533563 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:55.517285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.0.97.62:45351)
0924 03:09:55.517504 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:55.517511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:55.517928 (+ 417us) server_negotiation.cc:407] Connection header received
0924 03:09:55.518072 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:55.518078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:55.518164 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:55.518274 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:55.519618 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:55.520875 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:55.522232 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:55.522910 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:55.523079 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:55.527247 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:55.527255 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:55.527261 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:55.527305 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:55.530869 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:55.531525 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:55.531532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:55.531535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:55.531690 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:55.532050 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:55.532055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:55.532058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:55.532420 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:55.532690 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:55.532969 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 03:09:55.533200 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:09:55.537701 7997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:09:55.538410 7997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:55.545809 8136 tablet_service.cc:1467] Processing CreateTablet for tablet 808fac92ac394b728bf151d6d4f283e7 (DEFAULT_TABLE table=test-table [id=b61600dd5bb34e67868afd09bc03c20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:55.545934 8274 tablet_service.cc:1467] Processing CreateTablet for tablet 808fac92ac394b728bf151d6d4f283e7 (DEFAULT_TABLE table=test-table [id=b61600dd5bb34e67868afd09bc03c20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:55.546289 8136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 808fac92ac394b728bf151d6d4f283e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:55.546375 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 808fac92ac394b728bf151d6d4f283e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:55.551697 8584 tablet_bootstrap.cc:492] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14: Bootstrap starting.
I20260924 03:09:55.552301 8413 tablet_service.cc:1467] Processing CreateTablet for tablet 808fac92ac394b728bf151d6d4f283e7 (DEFAULT_TABLE table=test-table [id=b61600dd5bb34e67868afd09bc03c20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:55.552839 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 808fac92ac394b728bf151d6d4f283e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:55.553766 8584 tablet_bootstrap.cc:654] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:55.556850 8584 tablet_bootstrap.cc:492] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14: No bootstrap required, opened a new log
I20260924 03:09:55.557022 8584 ts_tablet_manager.cc:1397] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 03:09:55.557644 8584 raft_consensus.cc:348] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:55.557855 8584 raft_consensus.cc:374] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:55.557948 8584 raft_consensus.cc:729] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86caa800b6b04ddca55a6d9e558d8f14, State: Initialized, Role: FOLLOWER
I20260924 03:09:55.558229 8584 consensus_queue.cc:260] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:55.561934 8584 ts_tablet_manager.cc:1428] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260924 03:09:55.565097 8586 tablet_bootstrap.cc:492] T 808fac92ac394b728bf151d6d4f283e7 P e5e206871bfb44ee82ac6f04a7842f0e: Bootstrap starting.
I20260924 03:09:55.565645 8587 tablet_bootstrap.cc:492] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383: Bootstrap starting.
I20260924 03:09:55.567732 8586 tablet_bootstrap.cc:654] T 808fac92ac394b728bf151d6d4f283e7 P e5e206871bfb44ee82ac6f04a7842f0e: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:55.568226 8587 tablet_bootstrap.cc:654] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:55.570266 8586 tablet_bootstrap.cc:492] T 808fac92ac394b728bf151d6d4f283e7 P e5e206871bfb44ee82ac6f04a7842f0e: No bootstrap required, opened a new log
I20260924 03:09:55.570421 8586 ts_tablet_manager.cc:1397] T 808fac92ac394b728bf151d6d4f283e7 P e5e206871bfb44ee82ac6f04a7842f0e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 03:09:55.571231 8586 raft_consensus.cc:348] T 808fac92ac394b728bf151d6d4f283e7 P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:55.571492 8586 raft_consensus.cc:374] T 808fac92ac394b728bf151d6d4f283e7 P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:55.571592 8586 raft_consensus.cc:729] T 808fac92ac394b728bf151d6d4f283e7 P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e206871bfb44ee82ac6f04a7842f0e, State: Initialized, Role: FOLLOWER
I20260924 03:09:55.571703 8587 tablet_bootstrap.cc:492] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383: No bootstrap required, opened a new log
I20260924 03:09:55.571878 8587 ts_tablet_manager.cc:1397] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260924 03:09:55.571849 8586 consensus_queue.cc:260] T 808fac92ac394b728bf151d6d4f283e7 P e5e206871bfb44ee82ac6f04a7842f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:55.572461 8586 ts_tablet_manager.cc:1428] T 808fac92ac394b728bf151d6d4f283e7 P e5e206871bfb44ee82ac6f04a7842f0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 03:09:55.573639 8587 raft_consensus.cc:348] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:55.573971 8587 raft_consensus.cc:374] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:55.574090 8587 raft_consensus.cc:729] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6173798a8be148beaf8a858af433c383, State: Initialized, Role: FOLLOWER
I20260924 03:09:55.574380 8587 consensus_queue.cc:260] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:55.575515 8587 ts_tablet_manager.cc:1428] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260924 03:09:55.672691 8514 raft_consensus.cc:482] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:55.672991 8514 raft_consensus.cc:504] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:55.673735 8514 leader_election.cc:288] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86caa800b6b04ddca55a6d9e558d8f14 (127.0.97.1:45267), e5e206871bfb44ee82ac6f04a7842f0e (127.0.97.2:40267)
I20260924 03:09:55.674472 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "808fac92ac394b728bf151d6d4f283e7" candidate_uuid: "6173798a8be148beaf8a858af433c383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86caa800b6b04ddca55a6d9e558d8f14" is_pre_election: true
I20260924 03:09:55.674517 8295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "808fac92ac394b728bf151d6d4f283e7" candidate_uuid: "6173798a8be148beaf8a858af433c383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" is_pre_election: true
I20260924 03:09:55.674758 8156 raft_consensus.cc:2454] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6173798a8be148beaf8a858af433c383 in term 0.
I20260924 03:09:55.674786 8295 raft_consensus.cc:2454] T 808fac92ac394b728bf151d6d4f283e7 P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6173798a8be148beaf8a858af433c383 in term 0.
I20260924 03:09:55.675285 8368 leader_election.cc:302] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6173798a8be148beaf8a858af433c383, 86caa800b6b04ddca55a6d9e558d8f14; no voters:
I20260924 03:09:55.675597 8514 raft_consensus.cc:2781] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:55.675711 8514 raft_consensus.cc:482] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:55.675848 8514 raft_consensus.cc:3037] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:55.678936 8514 raft_consensus.cc:504] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:55.679646 8514 leader_election.cc:288] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 [CANDIDATE]: Term 1 election: Requested vote from peers 86caa800b6b04ddca55a6d9e558d8f14 (127.0.97.1:45267), e5e206871bfb44ee82ac6f04a7842f0e (127.0.97.2:40267)
I20260924 03:09:55.680181 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "808fac92ac394b728bf151d6d4f283e7" candidate_uuid: "6173798a8be148beaf8a858af433c383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86caa800b6b04ddca55a6d9e558d8f14"
I20260924 03:09:55.680241 8295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "808fac92ac394b728bf151d6d4f283e7" candidate_uuid: "6173798a8be148beaf8a858af433c383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e206871bfb44ee82ac6f04a7842f0e"
I20260924 03:09:55.680469 8295 raft_consensus.cc:3037] T 808fac92ac394b728bf151d6d4f283e7 P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:55.680470 8156 raft_consensus.cc:3037] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:55.683779 8156 raft_consensus.cc:2454] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6173798a8be148beaf8a858af433c383 in term 1.
I20260924 03:09:55.683779 8295 raft_consensus.cc:2454] T 808fac92ac394b728bf151d6d4f283e7 P e5e206871bfb44ee82ac6f04a7842f0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6173798a8be148beaf8a858af433c383 in term 1.
I20260924 03:09:55.684314 8368 leader_election.cc:302] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6173798a8be148beaf8a858af433c383, 86caa800b6b04ddca55a6d9e558d8f14; no voters:
I20260924 03:09:55.684777 8514 raft_consensus.cc:2781] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:55.685000 8514 raft_consensus.cc:686] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 [term 1 LEADER]: Becoming Leader. State: Replica: 6173798a8be148beaf8a858af433c383, State: Running, Role: LEADER
I20260924 03:09:55.685442 8514 consensus_queue.cc:237] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:55.688462 7998 catalog_manager.cc:5495] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6173798a8be148beaf8a858af433c383 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6173798a8be148beaf8a858af433c383" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } health_report { overall_health: HEALTHY } } }
I20260924 03:09:55.723745 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:55.718704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55872 (local address 127.0.97.3:38239)
0924 03:09:55.718882 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:55.718886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:55.719070 (+ 184us) server_negotiation.cc:407] Connection header received
0924 03:09:55.719147 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:55.719151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:55.719223 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:55.719316 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:55.719727 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:55.720568 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:55.722042 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:55.722472 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:55.722582 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:55.723092 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:55.723203 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:55.723273 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:55.723408 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 03:09:55.723470 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:09:55.727162 8156 raft_consensus.cc:1261] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14 [term 1 FOLLOWER]: Refusing update from remote peer 6173798a8be148beaf8a858af433c383: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:55.727162 8295 raft_consensus.cc:1261] T 808fac92ac394b728bf151d6d4f283e7 P e5e206871bfb44ee82ac6f04a7842f0e [term 1 FOLLOWER]: Refusing update from remote peer 6173798a8be148beaf8a858af433c383: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:55.727809 8554 consensus_queue.cc:1035] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:55.728223 8514 consensus_queue.cc:1035] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:55.733767 8552 mvcc.cc:205] Tried to move back new op lower bound from 7332738644890800128 to 7332738644730281984. Current Snapshot: MvccSnapshot[applied={T|T < 7332738644890800128 or (T in {7332738644890800128})}]
I20260924 03:09:55.739073 7998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:55.739429 7998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:55.743737 7998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107: Sending DeleteTablet for 3 replicas of tablet 808fac92ac394b728bf151d6d4f283e7
I20260924 03:09:55.744812 8136 tablet_service.cc:1514] Processing DeleteTablet for tablet 808fac92ac394b728bf151d6d4f283e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:55 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46930
I20260924 03:09:55.745018 8274 tablet_service.cc:1514] Processing DeleteTablet for tablet 808fac92ac394b728bf151d6d4f283e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:55 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57662
I20260924 03:09:55.745323 8570 tablet_replica.cc:330] T 808fac92ac394b728bf151d6d4f283e7 P e5e206871bfb44ee82ac6f04a7842f0e: stopping tablet replica
I20260924 03:09:55.745325 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet 808fac92ac394b728bf151d6d4f283e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:55 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55850
I20260924 03:09:55.745375 8135 tablet_service.cc:1514] Processing DeleteTablet for tablet 808fac92ac394b728bf151d6d4f283e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:55 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46930
I20260924 03:09:55.745536 8570 raft_consensus.cc:2229] T 808fac92ac394b728bf151d6d4f283e7 P e5e206871bfb44ee82ac6f04a7842f0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:55.745580 8569 tablet_replica.cc:330] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383: stopping tablet replica
I20260924 03:09:55.745639 8568 tablet_replica.cc:330] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14: stopping tablet replica
I20260924 03:09:55.745746 8570 raft_consensus.cc:2258] T 808fac92ac394b728bf151d6d4f283e7 P e5e206871bfb44ee82ac6f04a7842f0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:55.745776 8569 raft_consensus.cc:2229] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:09:55.745831 8568 raft_consensus.cc:2229] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:55.745970 8568 pending_rounds.cc:62] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14: Trying to abort 2 pending ops.
I20260924 03:09:55.746124 8569 raft_consensus.cc:2258] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:55.746089 8568 pending_rounds.cc:68] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7332738644890800128 op_type: WRITE_OP write_request { tablet_id: "808fac92ac394b728bf151d6d4f283e7" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\257\253\322\325\006\"8\n\ntest-admin\022*\n b61600dd5bb34e67868afd09bc03c20e\020\001\030\001 \001(\001" signature: "<redacted>""G\r\213\326\022\245\037\270\217\310\240\033%\220#0\350\222Sy)\204\221\224\033FKDF9\215\004X\370\017\325\263=0\312\240K#\355&\375z\375b 1J\302)s\222\372\030B\344\270\022\340*" signing_key_seq_num: 0 } } request_id { client_id: "83109d3ef7e7484287980c291090a310" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260924 03:09:55.746803 8568 pending_rounds.cc:68] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332738644730281984 op_type: NO_OP noop_request { }
I20260924 03:09:55.746963 8568 raft_consensus.cc:2866] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260924 03:09:55.747085 8568 raft_consensus.cc:2258] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:55.747117 8570 ts_tablet_manager.cc:1905] T 808fac92ac394b728bf151d6d4f283e7 P e5e206871bfb44ee82ac6f04a7842f0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:55.747061 7996 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:09:55.747961 7996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:55.748391 8568 ts_tablet_manager.cc:1905] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:55.753722 8570 ts_tablet_manager.cc:1918] T 808fac92ac394b728bf151d6d4f283e7 P e5e206871bfb44ee82ac6f04a7842f0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:55.753885 8570 log.cc:1196] T 808fac92ac394b728bf151d6d4f283e7 P e5e206871bfb44ee82ac6f04a7842f0e: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/808fac92ac394b728bf151d6d4f283e7
I20260924 03:09:55.754674 8135 tablet_service.cc:1467] Processing CreateTablet for tablet c6d59e55b59142c1a9cdbb494d265f8c (DEFAULT_TABLE table=test-table [id=42b42f68cf5c442586c7379f9e8fae9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:55.754894 8413 tablet_service.cc:1467] Processing CreateTablet for tablet c6d59e55b59142c1a9cdbb494d265f8c (DEFAULT_TABLE table=test-table [id=42b42f68cf5c442586c7379f9e8fae9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:55.755330 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d59e55b59142c1a9cdbb494d265f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:55.755297 8274 tablet_service.cc:1467] Processing CreateTablet for tablet c6d59e55b59142c1a9cdbb494d265f8c (DEFAULT_TABLE table=test-table [id=42b42f68cf5c442586c7379f9e8fae9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:55.755656 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d59e55b59142c1a9cdbb494d265f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:55.756780 8570 ts_tablet_manager.cc:1939] T 808fac92ac394b728bf151d6d4f283e7 P e5e206871bfb44ee82ac6f04a7842f0e: Deleting consensus metadata
I20260924 03:09:55.758083 7986 catalog_manager.cc:4841] TS e5e206871bfb44ee82ac6f04a7842f0e (127.0.97.2:40267): tablet 808fac92ac394b728bf151d6d4f283e7 (table test-table [id=b61600dd5bb34e67868afd09bc03c20e]) successfully deleted
I20260924 03:09:55.760772 8135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d59e55b59142c1a9cdbb494d265f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:55.762162 8587 tablet_bootstrap.cc:492] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383: Bootstrap starting.
I20260924 03:09:55.764009 8587 tablet_bootstrap.cc:654] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:55.764880 8568 ts_tablet_manager.cc:1918] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:55.765074 8568 log.cc:1196] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/808fac92ac394b728bf151d6d4f283e7
I20260924 03:09:55.765568 8568 ts_tablet_manager.cc:1939] T 808fac92ac394b728bf151d6d4f283e7 P 86caa800b6b04ddca55a6d9e558d8f14: Deleting consensus metadata
I20260924 03:09:55.766425 8587 tablet_bootstrap.cc:492] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383: No bootstrap required, opened a new log
I20260924 03:09:55.766603 8587 ts_tablet_manager.cc:1397] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 03:09:55.766739 7986 catalog_manager.cc:4841] TS 86caa800b6b04ddca55a6d9e558d8f14 (127.0.97.1:45267): tablet 808fac92ac394b728bf151d6d4f283e7 (table test-table [id=b61600dd5bb34e67868afd09bc03c20e]) successfully deleted
W20260924 03:09:55.767409 7986 catalog_manager.cc:4808] TS 86caa800b6b04ddca55a6d9e558d8f14 (127.0.97.1:45267): delete failed for tablet 808fac92ac394b728bf151d6d4f283e7 because the tablet was not found. No further retry: Not found: Tablet not found: 808fac92ac394b728bf151d6d4f283e7
I20260924 03:09:55.767726 8569 ts_tablet_manager.cc:1905] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:55.767477 8587 raft_consensus.cc:348] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:55.767901 8587 raft_consensus.cc:374] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:55.767999 8587 raft_consensus.cc:729] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6173798a8be148beaf8a858af433c383, State: Initialized, Role: FOLLOWER
I20260924 03:09:55.768322 8587 consensus_queue.cc:260] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:55.769025 8584 tablet_bootstrap.cc:492] T c6d59e55b59142c1a9cdbb494d265f8c P 86caa800b6b04ddca55a6d9e558d8f14: Bootstrap starting.
I20260924 03:09:55.769687 8587 ts_tablet_manager.cc:1428] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 03:09:55.770825 8584 tablet_bootstrap.cc:654] T c6d59e55b59142c1a9cdbb494d265f8c P 86caa800b6b04ddca55a6d9e558d8f14: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:55.773756 8586 tablet_bootstrap.cc:492] T c6d59e55b59142c1a9cdbb494d265f8c P e5e206871bfb44ee82ac6f04a7842f0e: Bootstrap starting.
I20260924 03:09:55.775096 8586 tablet_bootstrap.cc:654] T c6d59e55b59142c1a9cdbb494d265f8c P e5e206871bfb44ee82ac6f04a7842f0e: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:55.775550 8569 ts_tablet_manager.cc:1918] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:55.775704 8569 log.cc:1196] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/808fac92ac394b728bf151d6d4f283e7
I20260924 03:09:55.776150 8569 ts_tablet_manager.cc:1939] T 808fac92ac394b728bf151d6d4f283e7 P 6173798a8be148beaf8a858af433c383: Deleting consensus metadata
I20260924 03:09:55.776717 8584 tablet_bootstrap.cc:492] T c6d59e55b59142c1a9cdbb494d265f8c P 86caa800b6b04ddca55a6d9e558d8f14: No bootstrap required, opened a new log
I20260924 03:09:55.776912 8584 ts_tablet_manager.cc:1397] T c6d59e55b59142c1a9cdbb494d265f8c P 86caa800b6b04ddca55a6d9e558d8f14: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260924 03:09:55.777281 7985 catalog_manager.cc:4841] TS 6173798a8be148beaf8a858af433c383 (127.0.97.3:38239): tablet 808fac92ac394b728bf151d6d4f283e7 (table test-table [id=b61600dd5bb34e67868afd09bc03c20e]) successfully deleted
I20260924 03:09:55.777751 8586 tablet_bootstrap.cc:492] T c6d59e55b59142c1a9cdbb494d265f8c P e5e206871bfb44ee82ac6f04a7842f0e: No bootstrap required, opened a new log
I20260924 03:09:55.777927 8586 ts_tablet_manager.cc:1397] T c6d59e55b59142c1a9cdbb494d265f8c P e5e206871bfb44ee82ac6f04a7842f0e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260924 03:09:55.777806 8584 raft_consensus.cc:348] T c6d59e55b59142c1a9cdbb494d265f8c P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:55.778090 8584 raft_consensus.cc:374] T c6d59e55b59142c1a9cdbb494d265f8c P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:55.778173 8584 raft_consensus.cc:729] T c6d59e55b59142c1a9cdbb494d265f8c P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86caa800b6b04ddca55a6d9e558d8f14, State: Initialized, Role: FOLLOWER
I20260924 03:09:55.778429 8584 consensus_queue.cc:260] T c6d59e55b59142c1a9cdbb494d265f8c P 86caa800b6b04ddca55a6d9e558d8f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:55.778704 8586 raft_consensus.cc:348] T c6d59e55b59142c1a9cdbb494d265f8c P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:55.778867 8584 ts_tablet_manager.cc:1428] T c6d59e55b59142c1a9cdbb494d265f8c P 86caa800b6b04ddca55a6d9e558d8f14: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 03:09:55.778939 8586 raft_consensus.cc:374] T c6d59e55b59142c1a9cdbb494d265f8c P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:55.779027 8586 raft_consensus.cc:729] T c6d59e55b59142c1a9cdbb494d265f8c P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e206871bfb44ee82ac6f04a7842f0e, State: Initialized, Role: FOLLOWER
I20260924 03:09:55.779260 8586 consensus_queue.cc:260] T c6d59e55b59142c1a9cdbb494d265f8c P e5e206871bfb44ee82ac6f04a7842f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:55.779793 8586 ts_tablet_manager.cc:1428] T c6d59e55b59142c1a9cdbb494d265f8c P e5e206871bfb44ee82ac6f04a7842f0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 03:09:55.815904 8554 raft_consensus.cc:482] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:55.816087 8554 raft_consensus.cc:504] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:55.816716 8554 leader_election.cc:288] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86caa800b6b04ddca55a6d9e558d8f14 (127.0.97.1:45267), e5e206871bfb44ee82ac6f04a7842f0e (127.0.97.2:40267)
I20260924 03:09:55.817396 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d59e55b59142c1a9cdbb494d265f8c" candidate_uuid: "6173798a8be148beaf8a858af433c383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86caa800b6b04ddca55a6d9e558d8f14" is_pre_election: true
I20260924 03:09:55.817487 8295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d59e55b59142c1a9cdbb494d265f8c" candidate_uuid: "6173798a8be148beaf8a858af433c383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" is_pre_election: true
I20260924 03:09:55.817646 8156 raft_consensus.cc:2454] T c6d59e55b59142c1a9cdbb494d265f8c P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6173798a8be148beaf8a858af433c383 in term 0.
I20260924 03:09:55.817828 8295 raft_consensus.cc:2454] T c6d59e55b59142c1a9cdbb494d265f8c P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6173798a8be148beaf8a858af433c383 in term 0.
I20260924 03:09:55.818089 8368 leader_election.cc:302] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6173798a8be148beaf8a858af433c383, 86caa800b6b04ddca55a6d9e558d8f14; no voters:
I20260924 03:09:55.818389 8554 raft_consensus.cc:2781] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:55.818521 8554 raft_consensus.cc:482] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:55.818621 8554 raft_consensus.cc:3037] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:55.820941 8554 raft_consensus.cc:504] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:55.821518 8554 leader_election.cc:288] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 [CANDIDATE]: Term 1 election: Requested vote from peers 86caa800b6b04ddca55a6d9e558d8f14 (127.0.97.1:45267), e5e206871bfb44ee82ac6f04a7842f0e (127.0.97.2:40267)
I20260924 03:09:55.822039 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d59e55b59142c1a9cdbb494d265f8c" candidate_uuid: "6173798a8be148beaf8a858af433c383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86caa800b6b04ddca55a6d9e558d8f14"
I20260924 03:09:55.822065 8295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d59e55b59142c1a9cdbb494d265f8c" candidate_uuid: "6173798a8be148beaf8a858af433c383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e206871bfb44ee82ac6f04a7842f0e"
I20260924 03:09:55.822256 8156 raft_consensus.cc:3037] T c6d59e55b59142c1a9cdbb494d265f8c P 86caa800b6b04ddca55a6d9e558d8f14 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:55.822290 8295 raft_consensus.cc:3037] T c6d59e55b59142c1a9cdbb494d265f8c P e5e206871bfb44ee82ac6f04a7842f0e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:55.824800 8156 raft_consensus.cc:2454] T c6d59e55b59142c1a9cdbb494d265f8c P 86caa800b6b04ddca55a6d9e558d8f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6173798a8be148beaf8a858af433c383 in term 1.
I20260924 03:09:55.825120 8295 raft_consensus.cc:2454] T c6d59e55b59142c1a9cdbb494d265f8c P e5e206871bfb44ee82ac6f04a7842f0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6173798a8be148beaf8a858af433c383 in term 1.
I20260924 03:09:55.825182 8368 leader_election.cc:302] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6173798a8be148beaf8a858af433c383, 86caa800b6b04ddca55a6d9e558d8f14; no voters:
I20260924 03:09:55.825515 8554 raft_consensus.cc:2781] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:55.825659 8554 raft_consensus.cc:686] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 [term 1 LEADER]: Becoming Leader. State: Replica: 6173798a8be148beaf8a858af433c383, State: Running, Role: LEADER
I20260924 03:09:55.825932 8554 consensus_queue.cc:237] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } }
I20260924 03:09:55.828979 7996 catalog_manager.cc:5495] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6173798a8be148beaf8a858af433c383 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6173798a8be148beaf8a858af433c383" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6173798a8be148beaf8a858af433c383" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38239 } health_report { overall_health: HEALTHY } } }
I20260924 03:09:55.866415 8295 raft_consensus.cc:1261] T c6d59e55b59142c1a9cdbb494d265f8c P e5e206871bfb44ee82ac6f04a7842f0e [term 1 FOLLOWER]: Refusing update from remote peer 6173798a8be148beaf8a858af433c383: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:55.866408 8156 raft_consensus.cc:1261] T c6d59e55b59142c1a9cdbb494d265f8c P 86caa800b6b04ddca55a6d9e558d8f14 [term 1 FOLLOWER]: Refusing update from remote peer 6173798a8be148beaf8a858af433c383: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:55.866969 8554 consensus_queue.cc:1035] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86caa800b6b04ddca55a6d9e558d8f14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:55.867268 8514 consensus_queue.cc:1035] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e206871bfb44ee82ac6f04a7842f0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:55.915359 7998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:55.915591 7998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:55.919009 7998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1bb6f4bf97fe4d9c8b3ba9e986933107: Sending DeleteTablet for 3 replicas of tablet c6d59e55b59142c1a9cdbb494d265f8c
I20260924 03:09:55.919840 8135 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d59e55b59142c1a9cdbb494d265f8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:55 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46930
I20260924 03:09:55.919916 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d59e55b59142c1a9cdbb494d265f8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:55 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55850
I20260924 03:09:55.920231 8569 tablet_replica.cc:330] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383: stopping tablet replica
I20260924 03:09:55.920203 8274 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d59e55b59142c1a9cdbb494d265f8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:55 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57662
I20260924 03:09:55.920432 8569 raft_consensus.cc:2229] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:09:55.920527 8570 tablet_replica.cc:330] T c6d59e55b59142c1a9cdbb494d265f8c P e5e206871bfb44ee82ac6f04a7842f0e: stopping tablet replica
I20260924 03:09:55.920594 8568 tablet_replica.cc:330] T c6d59e55b59142c1a9cdbb494d265f8c P 86caa800b6b04ddca55a6d9e558d8f14: stopping tablet replica
I20260924 03:09:55.920694 8570 raft_consensus.cc:2229] T c6d59e55b59142c1a9cdbb494d265f8c P e5e206871bfb44ee82ac6f04a7842f0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:55.920812 8568 raft_consensus.cc:2229] T c6d59e55b59142c1a9cdbb494d265f8c P 86caa800b6b04ddca55a6d9e558d8f14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:55.920881 8570 raft_consensus.cc:2258] T c6d59e55b59142c1a9cdbb494d265f8c P e5e206871bfb44ee82ac6f04a7842f0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:55.920868 8569 raft_consensus.cc:2258] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:55.921042 8568 raft_consensus.cc:2258] T c6d59e55b59142c1a9cdbb494d265f8c P 86caa800b6b04ddca55a6d9e558d8f14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:55.922030 8568 ts_tablet_manager.cc:1905] T c6d59e55b59142c1a9cdbb494d265f8c P 86caa800b6b04ddca55a6d9e558d8f14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:55.922030 8569 ts_tablet_manager.cc:1905] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:55.923099 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 8070
I20260924 03:09:55.929270 8569 ts_tablet_manager.cc:1918] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:09:55.929431 8569 log.cc:1196] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/c6d59e55b59142c1a9cdbb494d265f8c
I20260924 03:09:55.929467 8568 ts_tablet_manager.cc:1918] T c6d59e55b59142c1a9cdbb494d265f8c P 86caa800b6b04ddca55a6d9e558d8f14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:09:55.929591 8568 log.cc:1196] T c6d59e55b59142c1a9cdbb494d265f8c P 86caa800b6b04ddca55a6d9e558d8f14: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/c6d59e55b59142c1a9cdbb494d265f8c
I20260924 03:09:55.929926 8569 ts_tablet_manager.cc:1939] T c6d59e55b59142c1a9cdbb494d265f8c P 6173798a8be148beaf8a858af433c383: Deleting consensus metadata
I20260924 03:09:55.930121 8568 ts_tablet_manager.cc:1939] T c6d59e55b59142c1a9cdbb494d265f8c P 86caa800b6b04ddca55a6d9e558d8f14: Deleting consensus metadata
I20260924 03:09:55.931306 7985 catalog_manager.cc:4841] TS 6173798a8be148beaf8a858af433c383 (127.0.97.3:38239): tablet c6d59e55b59142c1a9cdbb494d265f8c (table test-table [id=42b42f68cf5c442586c7379f9e8fae9d]) successfully deleted
I20260924 03:09:55.931973 8570 ts_tablet_manager.cc:1905] T c6d59e55b59142c1a9cdbb494d265f8c P e5e206871bfb44ee82ac6f04a7842f0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:55.936338 8570 ts_tablet_manager.cc:1918] T c6d59e55b59142c1a9cdbb494d265f8c P e5e206871bfb44ee82ac6f04a7842f0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:09:55.936450 8570 log.cc:1196] T c6d59e55b59142c1a9cdbb494d265f8c P e5e206871bfb44ee82ac6f04a7842f0e: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/c6d59e55b59142c1a9cdbb494d265f8c
I20260924 03:09:55.936925 8570 ts_tablet_manager.cc:1939] T c6d59e55b59142c1a9cdbb494d265f8c P e5e206871bfb44ee82ac6f04a7842f0e: Deleting consensus metadata
I20260924 03:09:55.936925 7986 catalog_manager.cc:4841] TS 86caa800b6b04ddca55a6d9e558d8f14 (127.0.97.1:45267): tablet c6d59e55b59142c1a9cdbb494d265f8c (table test-table [id=42b42f68cf5c442586c7379f9e8fae9d]) successfully deleted
I20260924 03:09:55.938053 7986 catalog_manager.cc:4841] TS e5e206871bfb44ee82ac6f04a7842f0e (127.0.97.2:40267): tablet c6d59e55b59142c1a9cdbb494d265f8c (table test-table [id=42b42f68cf5c442586c7379f9e8fae9d]) successfully deleted
Sep 24 03:09:55 dist-test-slave-5l98 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260924 03:09:55.950766 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:55.924141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46946 (local address 127.0.97.1:45267)
0924 03:09:55.936211 (+ 12070us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:55.936216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:55.936235 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:09:55.936289 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:55.936292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:55.936361 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:55.936544 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:55.938285 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:55.939085 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:55.940291 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:55.940773 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:55.940917 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:55.945004 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:55.945017 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:55.945032 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:55.945063 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:55.947958 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:55.948740 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:55.948747 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:55.948750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:55.948900 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:55.949262 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:55.949265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:55.949267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:55.949656 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:55.949880 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:55.950343 (+ 463us) server_negotiation.cc:299] Negotiation successful
0924 03:09:55.950506 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11940,"thread_start_us":12484,"threads_started":1}
I20260924 03:09:55.951435 8201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:56.051591 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 8070
I20260924 03:09:56.069705 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 8210
Sep 24 03:09:56 dist-test-slave-5l98 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260924 03:09:56.087419 8603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:56.070561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.0.97.2:40267)
0924 03:09:56.071095 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:56.071102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:56.071129 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:09:56.071183 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:56.071187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:56.071259 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:56.071391 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:56.072934 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:56.073763 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:56.075131 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:56.075638 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:56.075797 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:56.080440 (+ 4643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:56.080463 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:56.080480 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:56.080593 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:56.084015 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:56.085157 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:56.085164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:56.085167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:56.085309 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:56.085741 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:56.085745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:56.085750 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:56.086176 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:56.086493 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:56.086784 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 03:09:56.087049 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":242,"threads_started":1}
I20260924 03:09:56.088228 8340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:56.187445 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 8210
I20260924 03:09:56.209614 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 8349
Sep 24 03:09:56 dist-test-slave-5l98 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260924 03:09:56.226722 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:56.210479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48014 (local address 127.0.97.3:38239)
0924 03:09:56.210711 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:56.210716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:56.210734 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:09:56.210791 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:56.210794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:56.210857 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:56.210945 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:56.212224 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:56.212983 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:56.214224 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:56.214689 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:56.214795 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:56.219973 (+ 5178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:56.219996 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:56.220026 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:56.220142 (+ 116us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:56.223547 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:56.224367 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:56.224374 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:56.224376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:56.224568 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:56.225062 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:56.225066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:56.225068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:56.225581 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:56.225822 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:56.226198 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 03:09:56.226366 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 03:09:56.227440 8479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:56.376039 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 8349
I20260924 03:09:56.401157 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 7967
I20260924 03:09:56.402350 8039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:09:56.522295 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 7967
2026-09-24T03:09:56Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6158 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:09:56 dist-test-slave-5l98 krb5kdc[8609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:09:56 dist-test-slave-5l98 krb5kdc[8609](info): set up 2 sockets
Sep 24 03:09:56 dist-test-slave-5l98 krb5kdc[8609](info): commencing operation
krb5kdc: starting...
W20260924 03:09:57.665580 388 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:09:57 dist-test-slave-5l98 krb5kdc[8609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:09:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:09:57.906297 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45269
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:58.243893 8622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:58.244216 8622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:58.244305 8622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:58.251678 8622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:58.251787 8622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:58.251839 8622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:58.251899 8622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:58.251946 8622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:09:58.261868 8622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:09:58.263254 8622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:58.264532 8622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:58.281106 8630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:58.281083 8628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:58.281157 8627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:58.282225 8622 server_base.cc:1034] running on GCE node
I20260924 03:09:58.283047 8622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:58.284621 8622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:58.285853 8622 hybrid_clock.cc:648] HybridClock initialized: now 1790219398285802 us; error 47 us; skew 500 ppm
I20260924 03:09:58.286209 8622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T03:09:58Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1843 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:09:58 dist-test-slave-5l98 krb5kdc[8636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:09:58 dist-test-slave-5l98 krb5kdc[8636](info): set up 2 sockets
Sep 24 03:09:58 dist-test-slave-5l98 krb5kdc[8636](info): commencing operation
krb5kdc: starting...
W20260924 03:09:59.493572 388 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:09:59 dist-test-slave-5l98 krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:09:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:09:59.744732 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39463
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:00.092824 8649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:00.093153 8649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:00.093294 8649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:00.100756 8649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:10:00.100886 8649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:00.100940 8649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:00.101040 8649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:10:00.101126 8649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:10:00.111970 8649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:10:00.113417 8649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:00.114806 8649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:00.127444 8654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:00.127391 8657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:00.127467 8655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:00.128329 8649 server_base.cc:1034] running on GCE node
I20260924 03:10:00.129321 8649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:00.130837 8649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:00.132093 8649 hybrid_clock.cc:648] HybridClock initialized: now 1790219400132032 us; error 57 us; skew 500 ppm
I20260924 03:10:00.132575 8649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:10:00 dist-test-slave-5l98 krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:00.136094 8649 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:10:00.137846 8649 webserver.cc:460] Webserver started at http://127.0.97.62:35273/ using document root <none> and password file <none>
I20260924 03:10:00.138561 8649 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:00.138720 8649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:00.139124 8649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:00.142242 8649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "f983a5c97d8842bcadb9d6edf646ab52"
format_stamp: "Formatted at 2026-09-24 03:10:00 on dist-test-slave-5l98"
server_key: "ae60a40879d6dbc95970f8df59b059a1"
server_key_iv: "379b6f91508637cef576ca7e29c72ca3"
server_key_version: "encryptionkey@0"
I20260924 03:10:00.143038 8649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "f983a5c97d8842bcadb9d6edf646ab52"
format_stamp: "Formatted at 2026-09-24 03:10:00 on dist-test-slave-5l98"
server_key: "ae60a40879d6dbc95970f8df59b059a1"
server_key_iv: "379b6f91508637cef576ca7e29c72ca3"
server_key_version: "encryptionkey@0"
I20260924 03:10:00.148744 8649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 03:10:00.153160 8664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:00.154901 8649 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 03:10:00.155145 8649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "f983a5c97d8842bcadb9d6edf646ab52"
format_stamp: "Formatted at 2026-09-24 03:10:00 on dist-test-slave-5l98"
server_key: "ae60a40879d6dbc95970f8df59b059a1"
server_key_iv: "379b6f91508637cef576ca7e29c72ca3"
server_key_version: "encryptionkey@0"
I20260924 03:10:00.155369 8649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:00.174854 8649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:00.180840 8649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:00.181176 8649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:00.206611 8649 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39463
I20260924 03:10:00.206632 8715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39463 every 8 connection(s)
I20260924 03:10:00.208112 8649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:10:00.209455 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 8649
I20260924 03:10:00.209684 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:10:00.210510 388 external_mini_cluster.cc:1428] Setting key 844a8e2253fcf1e3735ad2f5739a738b
I20260924 03:10:00.213316 8716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 03:10:00 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:10:00.228148 8716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52: Bootstrap starting.
I20260924 03:10:00.232251 8716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:00.233506 8716 log.cc:824] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:00.237174 8719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:00.212791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42184 (local address 127.0.97.62:39463)
0924 03:10:00.213644 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:00.213655 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:00.213703 (+ 48us) server_negotiation.cc:407] Connection header received
0924 03:10:00.215291 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:00.215313 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:00.215703 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:00.216055 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:00.217916 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:00.219456 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:00.221929 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:00.223220 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:00.223461 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:00.228665 (+ 5204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:00.228675 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:00.228694 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:00.228746 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:00.232814 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:00.233618 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:00.233627 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:00.233650 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:00.233872 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:00.234273 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:00.234279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:00.234282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:00.234843 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:00.235154 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:00.235625 (+ 471us) server_negotiation.cc:299] Negotiation successful
0924 03:10:00.236053 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":365,"threads_started":1}
I20260924 03:10:00.237910 8716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52: No bootstrap required, opened a new log
I20260924 03:10:00.246309 8716 raft_consensus.cc:348] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f983a5c97d8842bcadb9d6edf646ab52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39463 } }
I20260924 03:10:00.246755 8716 raft_consensus.cc:374] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:00.246860 8716 raft_consensus.cc:729] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f983a5c97d8842bcadb9d6edf646ab52, State: Initialized, Role: FOLLOWER
I20260924 03:10:00.247470 8716 consensus_queue.cc:260] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f983a5c97d8842bcadb9d6edf646ab52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39463 } }
I20260924 03:10:00.247713 8716 raft_consensus.cc:388] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:10:00.247866 8716 raft_consensus.cc:482] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:10:00.248008 8716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:00.250800 8716 raft_consensus.cc:504] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f983a5c97d8842bcadb9d6edf646ab52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39463 } }
I20260924 03:10:00.251279 8716 leader_election.cc:302] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f983a5c97d8842bcadb9d6edf646ab52; no voters:
I20260924 03:10:00.251860 8716 leader_election.cc:288] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:10:00.252115 8721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:10:00.252956 8721 raft_consensus.cc:686] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [term 1 LEADER]: Becoming Leader. State: Replica: f983a5c97d8842bcadb9d6edf646ab52, State: Running, Role: LEADER
I20260924 03:10:00.253587 8721 consensus_queue.cc:237] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f983a5c97d8842bcadb9d6edf646ab52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39463 } }
I20260924 03:10:00.254083 8716 sys_catalog.cc:564] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:10:00.257256 8723 sys_catalog.cc:455] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f983a5c97d8842bcadb9d6edf646ab52. Latest consensus state: current_term: 1 leader_uuid: "f983a5c97d8842bcadb9d6edf646ab52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f983a5c97d8842bcadb9d6edf646ab52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39463 } } }
I20260924 03:10:00.257731 8723 sys_catalog.cc:458] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:00.258196 8722 sys_catalog.cc:455] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f983a5c97d8842bcadb9d6edf646ab52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f983a5c97d8842bcadb9d6edf646ab52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39463 } } }
I20260924 03:10:00.258540 8722 sys_catalog.cc:458] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:00.258891 8730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:10:00.265223 8730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:10:00.281634 8730 catalog_manager.cc:1347] Generated new cluster ID: 0a6753e0c4f54543a6fffc97ea56c708
I20260924 03:10:00.281791 8730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:10:00.324783 8730 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:10:00.326121 8730 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:10:00.340792 8730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52: Generated new TSK 0
I20260924 03:10:00.341679 8730 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 03:10:00.414896 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39463
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:00.755435 8742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:00.755739 8742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:00.755887 8742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:00.763942 8742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:00.764086 8742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:00.764294 8742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:10:00.774680 8742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:10:00.776086 8742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:00.777531 8742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:00.791950 8748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:00.791998 8747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:00.791905 8750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:00.792913 8742 server_base.cc:1034] running on GCE node
I20260924 03:10:00.793542 8742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:00.794466 8742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:00.795842 8742 hybrid_clock.cc:648] HybridClock initialized: now 1790219400795777 us; error 52 us; skew 500 ppm
I20260924 03:10:00.796273 8742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:10:00 dist-test-slave-5l98 krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:00.799811 8742 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:10:00.801434 8742 webserver.cc:460] Webserver started at http://127.0.97.1:36075/ using document root <none> and password file <none>
I20260924 03:10:00.802239 8742 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:00.802394 8742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:00.802721 8742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:00.805956 8742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "0034f4fea76145039642de24915d10cd"
format_stamp: "Formatted at 2026-09-24 03:10:00 on dist-test-slave-5l98"
server_key: "76883e83f848c6ab0f8d9cbd1d2d4eb5"
server_key_iv: "77476e604f5e23af79809c0fc5ed3839"
server_key_version: "encryptionkey@0"
I20260924 03:10:00.806975 8742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0034f4fea76145039642de24915d10cd"
format_stamp: "Formatted at 2026-09-24 03:10:00 on dist-test-slave-5l98"
server_key: "76883e83f848c6ab0f8d9cbd1d2d4eb5"
server_key_iv: "77476e604f5e23af79809c0fc5ed3839"
server_key_version: "encryptionkey@0"
I20260924 03:10:00.812777 8742 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:10:00.816602 8757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:00.818300 8742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:10:00.818511 8742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "0034f4fea76145039642de24915d10cd"
format_stamp: "Formatted at 2026-09-24 03:10:00 on dist-test-slave-5l98"
server_key: "76883e83f848c6ab0f8d9cbd1d2d4eb5"
server_key_iv: "77476e604f5e23af79809c0fc5ed3839"
server_key_version: "encryptionkey@0"
I20260924 03:10:00.818742 8742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:00.846395 8742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:00.851269 8742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:00.851665 8742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:00.852803 8742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:00.854323 8742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:00.854420 8742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:00.854564 8742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:00.854633 8742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:00.892170 8742 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40573
I20260924 03:10:00.892212 8869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40573 every 8 connection(s)
I20260924 03:10:00.893854 8742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:10:00.898355 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 8742
I20260924 03:10:00.898576 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:10:00.899511 388 external_mini_cluster.cc:1428] Setting key 5ca214a9d262ec8125a7b6973707649f
Sep 24 03:10:00 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:10:00.920842 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:00.896512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47309 (local address 127.0.97.62:39463)
0924 03:10:00.896947 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:00.896951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:00.898329 (+ 1378us) server_negotiation.cc:407] Connection header received
0924 03:10:00.901198 (+ 2869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:00.901204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:00.901275 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:00.901390 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:00.903813 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:00.904702 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:00.907707 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:00.908226 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:00.908346 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:00.912777 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:00.912786 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:00.912796 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:00.912845 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:00.917024 (+ 4179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:00.917862 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:00.917872 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:00.917878 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:00.918032 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:00.918487 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:00.918496 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:00.918502 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:00.918929 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:00.919112 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:00.919911 (+ 799us) server_negotiation.cc:299] Negotiation successful
0924 03:10:00.920214 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":220,"threads_started":1}
I20260924 03:10:00.921501 8873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:00.897126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.1:47309)
0924 03:10:00.898103 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:00.898165 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:00.900901 (+ 2736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:00.901768 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:00.901783 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:00.902636 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:00.903584 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:00.903616 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:00.904876 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:00.904884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:00.907365 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:00.907396 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:00.908535 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:00.908546 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:00.908938 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:00.909847 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:00.909891 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:00.912444 (+ 2553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:00.917216 (+ 4772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:00.917224 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:00.917240 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:00.917732 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:00.918171 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:00.918175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:00.918177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:00.918356 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:00.919118 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:00.919124 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:00.919478 (+ 354us) client_negotiation.cc:769] Sending connection context
0924 03:10:00.919762 (+ 284us) client_negotiation.cc:240] Negotiation successful
0924 03:10:00.920218 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":400,"threads_started":1}
I20260924 03:10:00.923374 8870 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39463
I20260924 03:10:00.923712 8870 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:00.924386 8870 heartbeater.cc:511] Master 127.0.97.62:39463 requested a full tablet report, sending...
I20260924 03:10:00.926463 8681 ts_manager.cc:194] Registered new tserver with Master: 0034f4fea76145039642de24915d10cd (127.0.97.1:40573)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 03:10:00.927970 8681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 03:10:00.956979 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39463
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:01.312121 8876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:01.312469 8876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:01.312659 8876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:01.320315 8876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:01.320477 8876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:01.320693 8876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:10:01.331439 8876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:10:01.333082 8876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:01.334451 8876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:01.348865 8881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:01.348901 8882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:01.349114 8884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:01.349982 8876 server_base.cc:1034] running on GCE node
I20260924 03:10:01.350698 8876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:01.351848 8876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:01.353134 8876 hybrid_clock.cc:648] HybridClock initialized: now 1790219401353067 us; error 52 us; skew 500 ppm
I20260924 03:10:01.353587 8876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:10:01 dist-test-slave-5l98 krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:01.357115 8876 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:10:01.358922 8876 webserver.cc:460] Webserver started at http://127.0.97.2:33511/ using document root <none> and password file <none>
I20260924 03:10:01.359798 8876 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:01.359907 8876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:01.360255 8876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:01.363385 8876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "33cabb7ca4a54f13985ef67ad8d17d72"
format_stamp: "Formatted at 2026-09-24 03:10:01 on dist-test-slave-5l98"
server_key: "9fd5ff9f4e2052c14e353e0d810b7014"
server_key_iv: "0c81070a12d2b09eea8a6a684035e099"
server_key_version: "encryptionkey@0"
I20260924 03:10:01.364149 8876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "33cabb7ca4a54f13985ef67ad8d17d72"
format_stamp: "Formatted at 2026-09-24 03:10:01 on dist-test-slave-5l98"
server_key: "9fd5ff9f4e2052c14e353e0d810b7014"
server_key_iv: "0c81070a12d2b09eea8a6a684035e099"
server_key_version: "encryptionkey@0"
I20260924 03:10:01.369743 8876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 03:10:01.373684 8891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:01.375231 8876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:10:01.375417 8876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "33cabb7ca4a54f13985ef67ad8d17d72"
format_stamp: "Formatted at 2026-09-24 03:10:01 on dist-test-slave-5l98"
server_key: "9fd5ff9f4e2052c14e353e0d810b7014"
server_key_iv: "0c81070a12d2b09eea8a6a684035e099"
server_key_version: "encryptionkey@0"
I20260924 03:10:01.375614 8876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:01.399559 8876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:01.404162 8876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:01.404464 8876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:01.405705 8876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:01.407109 8876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:01.407186 8876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:01.407325 8876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:01.407374 8876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:01.442484 8876 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40277
I20260924 03:10:01.442584 9003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40277 every 8 connection(s)
I20260924 03:10:01.443993 8876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:10:01.450817 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 8876
I20260924 03:10:01.450990 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:10:01.451448 388 external_mini_cluster.cc:1428] Setting key b5ffd5b5640a78eb641f1427ab215a3e
Sep 24 03:10:01 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:10:01.465430 9006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:01.446699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57997 (local address 127.0.97.62:39463)
0924 03:10:01.447145 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:01.447149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:01.448046 (+ 897us) server_negotiation.cc:407] Connection header received
0924 03:10:01.449986 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:01.449992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:01.450092 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:01.450207 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:01.451855 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:01.452785 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:01.454759 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:01.455222 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:01.455372 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:01.459370 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:01.459375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:01.459378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:01.459411 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:01.462324 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:01.463106 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:01.463112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:01.463115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:01.463243 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:01.463669 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:01.463674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:01.463677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:01.463946 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:01.464106 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:01.464828 (+ 722us) server_negotiation.cc:299] Negotiation successful
0924 03:10:01.465040 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":247,"threads_started":1}
I20260924 03:10:01.466437 9007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:01.447153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.2:57997)
0924 03:10:01.447883 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:01.447930 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:01.449710 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:01.450439 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:01.450449 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:01.451063 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:01.451688 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:01.451707 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:01.452934 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:01.452938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:01.454581 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:01.454610 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:01.455463 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:01.455470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:01.455722 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:01.456430 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:01.456454 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:01.459077 (+ 2623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:01.462469 (+ 3392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:01.462488 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:01.462502 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:01.462973 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:01.463355 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:01.463360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:01.463364 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:01.463546 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:01.464072 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:01.464080 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:01.464423 (+ 343us) client_negotiation.cc:769] Sending connection context
0924 03:10:01.464757 (+ 334us) client_negotiation.cc:240] Negotiation successful
0924 03:10:01.465294 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":281,"threads_started":1}
I20260924 03:10:01.468189 9004 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39463
I20260924 03:10:01.468537 9004 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:01.469283 9004 heartbeater.cc:511] Master 127.0.97.62:39463 requested a full tablet report, sending...
I20260924 03:10:01.470884 8681 ts_manager.cc:194] Registered new tserver with Master: 33cabb7ca4a54f13985ef67ad8d17d72 (127.0.97.2:40277)
I20260924 03:10:01.471830 8681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57997
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 03:10:01.507663 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39463
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:01.849022 9010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:01.849318 9010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:01.849411 9010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:01.857146 9010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:01.857312 9010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:01.857455 9010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:10:01.868140 9010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:10:01.870005 9010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:01.871659 9010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:01.886763 9015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:01.886790 9016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:01.886824 9018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:01.887974 9010 server_base.cc:1034] running on GCE node
I20260924 03:10:01.888895 9010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:01.890019 9010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:01.891273 9010 hybrid_clock.cc:648] HybridClock initialized: now 1790219401891206 us; error 47 us; skew 500 ppm
I20260924 03:10:01.891633 9010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:10:01 dist-test-slave-5l98 krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:01.895781 9010 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:10:01.898272 9010 webserver.cc:460] Webserver started at http://127.0.97.3:37729/ using document root <none> and password file <none>
I20260924 03:10:01.899060 9010 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:01.899190 9010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:01.899515 9010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:01.902727 9010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "37c3821d5016485c869f0751acdeb2fa"
format_stamp: "Formatted at 2026-09-24 03:10:01 on dist-test-slave-5l98"
server_key: "a26725fd2138277e771d82dcdc39c022"
server_key_iv: "3f2dc6fd39f2ce016fd2aa5e601e8898"
server_key_version: "encryptionkey@0"
I20260924 03:10:01.903635 9010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "37c3821d5016485c869f0751acdeb2fa"
format_stamp: "Formatted at 2026-09-24 03:10:01 on dist-test-slave-5l98"
server_key: "a26725fd2138277e771d82dcdc39c022"
server_key_iv: "3f2dc6fd39f2ce016fd2aa5e601e8898"
server_key_version: "encryptionkey@0"
I20260924 03:10:01.909622 9010 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 03:10:01.913847 9025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:01.915400 9010 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 03:10:01.915578 9010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "37c3821d5016485c869f0751acdeb2fa"
format_stamp: "Formatted at 2026-09-24 03:10:01 on dist-test-slave-5l98"
server_key: "a26725fd2138277e771d82dcdc39c022"
server_key_iv: "3f2dc6fd39f2ce016fd2aa5e601e8898"
server_key_version: "encryptionkey@0"
I20260924 03:10:01.915802 9010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:01.931058 8870 heartbeater.cc:503] Master 127.0.97.62:39463 was elected leader, sending a full tablet report...
I20260924 03:10:01.940948 9010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:01.945915 9010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:01.946259 9010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:01.947261 9010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:01.948701 9010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:01.948776 9010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:01.949018 9010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:01.949110 9010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:01.985245 9010 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38195
I20260924 03:10:01.985337 9137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38195 every 8 connection(s)
I20260924 03:10:01.986555 9010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:10:01.990897 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 9010
I20260924 03:10:01.991066 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:10:01.991487 388 external_mini_cluster.cc:1428] Setting key 884d0fd70b120d545d37a8f6f613ea08
Sep 24 03:10:01 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:10:02.009073 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:01.989499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44285 (local address 127.0.97.62:39463)
0924 03:10:01.989991 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:01.989995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:01.990814 (+ 819us) server_negotiation.cc:407] Connection header received
0924 03:10:01.993188 (+ 2374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:01.993193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:01.993302 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:01.993418 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:01.995372 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:01.996130 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:01.997930 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:01.998329 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:01.998471 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:02.002357 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:02.002362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:02.002365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:02.002399 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:02.005423 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:02.006318 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:02.006324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:02.006326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:02.006458 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:02.007162 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:02.007167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:02.007169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:02.007462 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:02.007627 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:02.008382 (+ 755us) server_negotiation.cc:299] Negotiation successful
0924 03:10:02.008655 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":273,"threads_started":1}
I20260924 03:10:02.010012 9141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:01.989869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.3:44285)
0924 03:10:01.990643 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:01.990681 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:01.992788 (+ 2107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:01.993663 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:01.993672 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:01.994472 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:01.995182 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:01.995209 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:01.996281 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:01.996287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:01.997776 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:01.997788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:01.998550 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:01.998557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:01.998816 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:01.999498 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:01.999522 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:02.002127 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:02.005585 (+ 3458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:02.005596 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:02.005621 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:02.006187 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:02.006569 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:02.006574 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:02.006578 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:02.006772 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:02.007663 (+ 891us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:02.007671 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:02.008006 (+ 335us) client_negotiation.cc:769] Sending connection context
0924 03:10:02.008286 (+ 280us) client_negotiation.cc:240] Negotiation successful
0924 03:10:02.008857 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":304,"threads_started":1}
I20260924 03:10:02.011556 9138 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39463
I20260924 03:10:02.011819 9138 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:02.012535 9138 heartbeater.cc:511] Master 127.0.97.62:39463 requested a full tablet report, sending...
I20260924 03:10:02.014082 8681 ts_manager.cc:194] Registered new tserver with Master: 37c3821d5016485c869f0751acdeb2fa (127.0.97.3:38195)
I20260924 03:10:02.014966 8681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44285
I20260924 03:10:02.019994 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260924 03:10:02.176856 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 8742
Sep 24 03:10:02 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:10:02.195916 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:02.178272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37876 (local address 127.0.97.1:40573)
0924 03:10:02.178987 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:02.178994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:02.179023 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:10:02.179086 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:02.179093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:02.179377 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:02.179529 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:02.181142 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:02.182231 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:02.183637 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:02.184268 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:02.184374 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:02.189048 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:02.189063 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:02.189079 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:02.189120 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:02.192128 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:02.193247 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:02.193259 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:02.193266 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:02.193449 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:02.193913 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:02.193920 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:02.193926 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:02.194418 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:02.194747 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:02.195087 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 03:10:02.195393 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":401,"threads_started":1}
I20260924 03:10:02.197854 8865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:10:02.286859 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 8742
I20260924 03:10:02.304383 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 8876
Sep 24 03:10:02 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 03:10:02.321780 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:02.305454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34038 (local address 127.0.97.2:40277)
0924 03:10:02.306139 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:02.306145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:02.306171 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:10:02.306230 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:02.306236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:02.306439 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:02.306576 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:02.307929 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:02.309241 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:02.310651 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:02.311223 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:02.311327 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:02.315638 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:02.315652 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:02.315667 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:02.315714 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:02.318650 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:02.319360 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:02.319367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:02.319369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:02.319493 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:02.319931 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:02.319935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:02.319938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:02.320416 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:02.320918 (+ 502us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:02.321167 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 03:10:02.321374 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":346,"threads_started":1}
I20260924 03:10:02.323052 8999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:10:02.408460 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 8876
I20260924 03:10:02.425169 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 9010
Sep 24 03:10:02 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 03:10:02.442574 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:02.426188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39742 (local address 127.0.97.3:38195)
0924 03:10:02.426776 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:02.426783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:02.426814 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:10:02.426892 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:02.426899 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:02.427145 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:02.427292 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:02.428696 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:02.429818 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:02.431192 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:02.431783 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:02.431899 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:02.436760 (+ 4861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:02.436773 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:02.436788 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:02.436830 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:02.439781 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:02.440455 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:02.440462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:02.440465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:02.440631 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:02.440955 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:02.440959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:02.440961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:02.441302 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:02.441518 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:02.441819 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 03:10:02.442182 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":306,"threads_started":1}
I20260924 03:10:02.443862 9133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:10:02.529763 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 9010
I20260924 03:10:02.546134 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 8649
I20260924 03:10:02.547516 8711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:10:02.651013 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 8649
I20260924 03:10:02.674605 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39463
--webserver_interface=127.0.97.62
--webserver_port=35273
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:03.013548 9156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:03.013865 9156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:03.013976 9156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:03.021413 9156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:10:03.021538 9156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:03.021584 9156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:03.021633 9156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:10:03.021669 9156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:10:03.031662 9156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:10:03.032974 9156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:03.034255 9156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:03.046371 9162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:03.046480 9161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:03.046480 9164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:03.047451 9156 server_base.cc:1034] running on GCE node
I20260924 03:10:03.048233 9156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:03.049782 9156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:03.051031 9156 hybrid_clock.cc:648] HybridClock initialized: now 1790219403050970 us; error 69 us; skew 500 ppm
I20260924 03:10:03.051429 9156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:10:03 dist-test-slave-5l98 krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:03.054837 9156 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:10:03.056551 9156 webserver.cc:460] Webserver started at http://127.0.97.62:35273/ using document root <none> and password file <none>
I20260924 03:10:03.057271 9156 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:03.057422 9156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:03.063036 9156 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 03:10:03.066303 9171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:03.067919 9156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:10:03.068143 9156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "f983a5c97d8842bcadb9d6edf646ab52"
format_stamp: "Formatted at 2026-09-24 03:10:00 on dist-test-slave-5l98"
server_key: "ae60a40879d6dbc95970f8df59b059a1"
server_key_iv: "379b6f91508637cef576ca7e29c72ca3"
server_key_version: "encryptionkey@0"
I20260924 03:10:03.069267 9156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:03.095947 9156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:03.100733 9156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:03.101055 9156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:03.122251 9156 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39463
I20260924 03:10:03.122306 9222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39463 every 8 connection(s)
I20260924 03:10:03.123837 9156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:10:03.128008 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 9156
I20260924 03:10:03.129091 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40573
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39463
--webserver_port=36075
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:10:03.133297 9223 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:10:03.138417 9223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52: Bootstrap starting.
I20260924 03:10:03.158447 9223 log.cc:824] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:03.167426 9223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:10:03.168002 9223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52: Bootstrap complete.
I20260924 03:10:03.177945 9223 raft_consensus.cc:348] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f983a5c97d8842bcadb9d6edf646ab52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39463 } }
I20260924 03:10:03.178834 9223 raft_consensus.cc:729] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f983a5c97d8842bcadb9d6edf646ab52, State: Initialized, Role: FOLLOWER
I20260924 03:10:03.179477 9223 consensus_queue.cc:260] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f983a5c97d8842bcadb9d6edf646ab52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39463 } }
I20260924 03:10:03.179736 9223 raft_consensus.cc:388] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:10:03.179853 9223 raft_consensus.cc:482] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:10:03.180034 9223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:10:03.183923 9223 raft_consensus.cc:504] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f983a5c97d8842bcadb9d6edf646ab52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39463 } }
I20260924 03:10:03.184479 9223 leader_election.cc:302] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f983a5c97d8842bcadb9d6edf646ab52; no voters:
I20260924 03:10:03.185220 9223 leader_election.cc:288] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 03:10:03.185350 9227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:10:03.186709 9227 raft_consensus.cc:686] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [term 2 LEADER]: Becoming Leader. State: Replica: f983a5c97d8842bcadb9d6edf646ab52, State: Running, Role: LEADER
I20260924 03:10:03.187299 9227 consensus_queue.cc:237] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f983a5c97d8842bcadb9d6edf646ab52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39463 } }
I20260924 03:10:03.187593 9223 sys_catalog.cc:564] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:10:03.190024 9228 sys_catalog.cc:455] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f983a5c97d8842bcadb9d6edf646ab52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f983a5c97d8842bcadb9d6edf646ab52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39463 } } }
I20260924 03:10:03.190471 9228 sys_catalog.cc:458] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:03.191097 9229 sys_catalog.cc:455] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f983a5c97d8842bcadb9d6edf646ab52. Latest consensus state: current_term: 2 leader_uuid: "f983a5c97d8842bcadb9d6edf646ab52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f983a5c97d8842bcadb9d6edf646ab52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39463 } } }
I20260924 03:10:03.191495 9229 sys_catalog.cc:458] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52 [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:03.195082 9233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:10:03.201334 9233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:10:03.202747 9233 catalog_manager.cc:1259] Loaded cluster ID: 0a6753e0c4f54543a6fffc97ea56c708
I20260924 03:10:03.202907 9233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:10:03.207118 9233 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:10:03.209192 9233 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f983a5c97d8842bcadb9d6edf646ab52: Loaded TSK: 0
I20260924 03:10:03.210775 9233 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:10:03.492074 9225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:03.492414 9225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:03.492605 9225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:03.500268 9225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:03.500420 9225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:03.500627 9225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:10:03.510601 9225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36075
--tserver_master_addrs=127.0.97.62:39463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:10:03.511967 9225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:03.513394 9225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:03.528825 9253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:03.529076 9250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:03.529227 9251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:03.529995 9225 server_base.cc:1034] running on GCE node
I20260924 03:10:03.530685 9225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:03.531710 9225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:03.532940 9225 hybrid_clock.cc:648] HybridClock initialized: now 1790219403532897 us; error 40 us; skew 500 ppm
I20260924 03:10:03.533310 9225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:10:03 dist-test-slave-5l98 krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219403, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:03.547892 9225 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260924 03:10:03.549789 9225 webserver.cc:460] Webserver started at http://127.0.97.1:36075/ using document root <none> and password file <none>
I20260924 03:10:03.550534 9225 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:03.550689 9225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:03.556577 9225 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:10:03.559537 9260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:03.561043 9225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:10:03.561188 9225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "0034f4fea76145039642de24915d10cd"
format_stamp: "Formatted at 2026-09-24 03:10:00 on dist-test-slave-5l98"
server_key: "76883e83f848c6ab0f8d9cbd1d2d4eb5"
server_key_iv: "77476e604f5e23af79809c0fc5ed3839"
server_key_version: "encryptionkey@0"
I20260924 03:10:03.562137 9225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:03.584278 9225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:03.588672 9225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:03.589001 9225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:03.589963 9225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:03.591305 9225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:03.591395 9225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:03.591470 9225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:03.591524 9225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:03.625605 9225 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40573
I20260924 03:10:03.625710 9372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40573 every 8 connection(s)
I20260924 03:10:03.627101 9225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:10:03.635666 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 9225
I20260924 03:10:03.636472 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40277
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39463
--webserver_port=33511
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:10:03 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:03 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 03:10:03.648743 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:03.629996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57927 (local address 127.0.97.62:39463)
0924 03:10:03.630681 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:03.630692 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:03.631092 (+ 400us) server_negotiation.cc:407] Connection header received
0924 03:10:03.633232 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:03.633245 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:03.633615 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:03.634140 (+ 525us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:03.635946 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.638213 (+ 2267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.639966 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.641143 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.641351 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:03.647588 (+ 6237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57927: BlockingRecv error: recv got EOF from 127.0.97.1:57927 (error 108)
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":275,"threads_started":1}
I20260924 03:10:03.648784 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:03.630027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.1:57927)
0924 03:10:03.630867 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:03.630919 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:03.632713 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:03.634537 (+ 1824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:03.634552 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:03.635171 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:03.635712 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:03.635729 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.638423 (+ 2694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.638433 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:03.639798 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:03.639812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.641473 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.641481 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:03.641756 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:03.642459 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:03.642485 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:03.647171 (+ 4686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":355,"threads_started":1}
W20260924 03:10:03.649700 9373 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39463 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:39463: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 24 03:10:03 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:03 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:03.659861 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:03.650401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.1:35885)
0924 03:10:03.650599 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:03.650622 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:03.650747 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:03.651186 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:03.651190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:03.651522 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:03.651670 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:03.651679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.653253 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.653258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:03.654169 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:03.654180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.655254 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.655259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:03.655394 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:03.656074 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:03.656093 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:03.659465 (+ 3372us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 03:10:03.660279 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:03.650628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35885 (local address 127.0.97.62:39463)
0924 03:10:03.650833 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:03.650842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:03.650866 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:10:03.650930 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:03.650938 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:03.651020 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:03.651127 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:03.651827 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.653102 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.654317 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.655062 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.655226 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:03.659877 (+ 4651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35885: BlockingRecv error: recv got EOF from 127.0.97.1:35885 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 03:10:03 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:03 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:03.669373 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:03.660822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.1:37579)
0924 03:10:03.661009 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:03.661032 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:03.661167 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:03.661494 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:03.661499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:03.661828 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:03.661974 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:03.661982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.662961 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.662966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:03.663911 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:03.663922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.664619 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.664625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:03.664802 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:03.665449 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:03.665537 (+ 88us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:03.668963 (+ 3426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":77}
W20260924 03:10:03.669718 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:03.660809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37579 (local address 127.0.97.62:39463)
0924 03:10:03.660967 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:03.660974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:03.661097 (+ 123us) server_negotiation.cc:407] Connection header received
0924 03:10:03.661287 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:03.661292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:03.661372 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:03.661470 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:03.662109 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.662817 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.664054 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.664458 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.664617 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:03.669249 (+ 4632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37579: BlockingRecv error: recv got EOF from 127.0.97.1:37579 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 03:10:03.669880 9373 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:10:03.980093 9377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:03.980432 9377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:03.980616 9377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:03.988023 9377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:03.988190 9377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:03.988356 9377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:10:03.998648 9377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33511
--tserver_master_addrs=127.0.97.62:39463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:10:04.000041 9377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:04.001344 9377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:04.015275 9385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:04.015326 9383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:04.015553 9382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:04.016788 9377 server_base.cc:1034] running on GCE node
I20260924 03:10:04.017500 9377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:04.018635 9377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:04.019862 9377 hybrid_clock.cc:648] HybridClock initialized: now 1790219404019756 us; error 101 us; skew 500 ppm
I20260924 03:10:04.020187 9377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:10:04 dist-test-slave-5l98 krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219404, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:04.023499 9377 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260924 03:10:04.025061 9377 webserver.cc:460] Webserver started at http://127.0.97.2:33511/ using document root <none> and password file <none>
I20260924 03:10:04.025728 9377 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:04.025839 9377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:04.031553 9377 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 03:10:04.034636 9392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:04.036098 9377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:10:04.036273 9377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "33cabb7ca4a54f13985ef67ad8d17d72"
format_stamp: "Formatted at 2026-09-24 03:10:01 on dist-test-slave-5l98"
server_key: "9fd5ff9f4e2052c14e353e0d810b7014"
server_key_iv: "0c81070a12d2b09eea8a6a684035e099"
server_key_version: "encryptionkey@0"
I20260924 03:10:04.037400 9377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:04.091053 9377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:04.095582 9377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:04.095824 9377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:04.096851 9377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:04.098235 9377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:04.098362 9377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:04.098488 9377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:04.098572 9377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:04.133227 9377 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40277
I20260924 03:10:04.133270 9504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40277 every 8 connection(s)
I20260924 03:10:04.134750 9377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:10:04.144953 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 9377
I20260924 03:10:04.145643 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:38195
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39463
--webserver_port=37729
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:10:04 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:04 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 03:10:04.152566 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:04.137531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41147 (local address 127.0.97.62:39463)
0924 03:10:04.137776 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:04.137780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:04.139260 (+ 1480us) server_negotiation.cc:407] Connection header received
0924 03:10:04.141166 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:04.141172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:04.141268 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:04.141378 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:04.142970 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.143673 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.145421 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.145912 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.146047 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:04.152179 (+ 6132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41147: BlockingRecv error: recv got EOF from 127.0.97.2:41147 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
I20260924 03:10:04.153254 9507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:04.138208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.2:41147)
0924 03:10:04.138987 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:04.139119 (+ 132us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:04.140865 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:04.141647 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:04.141655 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:04.142194 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:04.142787 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:04.142805 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.143829 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.143835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:04.145254 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:04.145265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.146150 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.146157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:04.146381 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:04.147204 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:04.147234 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:04.151940 (+ 4706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":321,"threads_started":1}
W20260924 03:10:04.154232 9505 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39463 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:39463: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 24 03:10:04 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:04 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:04.165997 9507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:04.154885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.2:33771)
0924 03:10:04.155075 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:04.155097 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:04.155217 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:04.155631 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:04.155639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:04.156120 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:04.156378 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:04.156392 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.157584 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.157592 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:04.159290 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:04.159308 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.160219 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.160228 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:04.160460 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:04.161246 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:04.161276 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:04.165560 (+ 4284us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 03:10:04.166065 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:04.155122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33771 (local address 127.0.97.62:39463)
0924 03:10:04.155291 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:04.155296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:04.155313 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:10:04.155364 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:04.155367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:04.155483 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:04.155567 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:04.156613 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.157422 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.159469 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.160035 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.160176 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:04.165794 (+ 5618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33771: BlockingRecv error: recv got EOF from 127.0.97.2:33771 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 03:10:04 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:04 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:04.176877 9507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:04.166969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.2:51061)
0924 03:10:04.167264 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:04.167286 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:04.167440 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:04.167830 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:04.167835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:04.168298 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:04.168681 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:04.168698 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.169814 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.169822 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:04.171623 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:04.171645 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.172656 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.172663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:04.172848 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:04.173556 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:04.173602 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:04.176527 (+ 2925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148}
W20260924 03:10:04.176985 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:04.167064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51061 (local address 127.0.97.62:39463)
0924 03:10:04.167237 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:04.167242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:04.167536 (+ 294us) server_negotiation.cc:407] Connection header received
0924 03:10:04.167613 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:04.167617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:04.167678 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:04.167758 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:04.168878 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.169659 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.171819 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.172422 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.172621 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:04.176756 (+ 4135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51061: BlockingRecv error: recv got EOF from 127.0.97.2:51061 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 03:10:04.177276 9505 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:10:04.516870 9508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:04.517251 9508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:04.517421 9508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:04.525192 9508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:04.525337 9508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:04.525521 9508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:10:04.535557 9508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:38195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37729
--tserver_master_addrs=127.0.97.62:39463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:10:04.536978 9508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:04.538266 9508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:04.552270 9514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:04.552302 9516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:04.552270 9513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:04.553107 9508 server_base.cc:1034] running on GCE node
I20260924 03:10:04.553750 9508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:04.554836 9508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:04.556365 9508 hybrid_clock.cc:648] HybridClock initialized: now 1790219404556283 us; error 67 us; skew 500 ppm
I20260924 03:10:04.556841 9508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:10:04 dist-test-slave-5l98 krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219404, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:04.560299 9508 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260924 03:10:04.562160 9508 webserver.cc:460] Webserver started at http://127.0.97.3:37729/ using document root <none> and password file <none>
I20260924 03:10:04.562868 9508 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:04.563024 9508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:04.568914 9508 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:10:04.572062 9523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:04.573746 9508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:10:04.573941 9508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "37c3821d5016485c869f0751acdeb2fa"
format_stamp: "Formatted at 2026-09-24 03:10:01 on dist-test-slave-5l98"
server_key: "a26725fd2138277e771d82dcdc39c022"
server_key_iv: "3f2dc6fd39f2ce016fd2aa5e601e8898"
server_key_version: "encryptionkey@0"
I20260924 03:10:04.575263 9508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:04.591022 9508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:04.595872 9508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:04.596186 9508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:04.597400 9508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:04.598796 9508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:04.598920 9508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:04.599044 9508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:04.599143 9508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:04.634495 9508 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38195
I20260924 03:10:04.634554 9635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38195 every 8 connection(s)
I20260924 03:10:04.635886 9508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:10:04.644984 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 9508
Sep 24 03:10:04 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:04 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 03:10:04.655532 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:04.638777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57127 (local address 127.0.97.62:39463)
0924 03:10:04.639016 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:04.639022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:04.640274 (+ 1252us) server_negotiation.cc:407] Connection header received
0924 03:10:04.642479 (+ 2205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:04.642484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:04.642577 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:04.642673 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:04.644467 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.645606 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.648211 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.648878 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.649227 (+ 349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:04.655202 (+ 5975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57127: BlockingRecv error: recv got EOF from 127.0.97.3:57127 (error 108)
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":80}
I20260924 03:10:04.656032 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:04.639318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.3:57127)
0924 03:10:04.640085 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:04.640140 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:04.642205 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:04.642937 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:04.642949 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:04.643623 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:04.644262 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:04.644280 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.645808 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.645840 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:04.648019 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:04.648039 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.649454 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.649460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:04.649698 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:04.650500 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:04.650530 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:04.654952 (+ 4422us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":336,"threads_started":1}
W20260924 03:10:04.656765 9636 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39463 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:39463: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260924 03:10:04.663628 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:04.646329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.0.97.62:39463)
0924 03:10:04.646843 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:04.646846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:04.646861 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:10:04.646902 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:04.646905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:04.646963 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:04.647049 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:04.648902 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.649981 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.651816 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.652422 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.652609 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:04.655881 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:04.655888 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:04.655904 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:04.655941 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:04.659517 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:04.660355 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:04.660362 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:04.660369 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:04.660662 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:04.661236 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:04.661244 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:04.661250 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:04.661940 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:04.662347 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:04.662844 (+ 497us) server_negotiation.cc:299] Negotiation successful
0924 03:10:04.663242 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":431,"thread_start_us":306,"threads_started":1}
Sep 24 03:10:04 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:04 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:04.667990 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:04.657341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.3:49521)
0924 03:10:04.657490 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:04.657512 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:04.657614 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:04.657978 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:04.657982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:04.658311 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:04.658453 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:04.658461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.659884 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.659891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:04.661403 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:04.661425 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.662548 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.662557 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:04.662760 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:04.663599 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:04.663618 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:04.667683 (+ 4065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 03:10:04.668138 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:04.657440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49521 (local address 127.0.97.62:39463)
0924 03:10:04.657619 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:04.657622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:04.657637 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:10:04.657728 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:04.657735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:04.657819 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:04.657924 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:04.658657 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.659758 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.661546 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.662363 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.662485 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:04.667873 (+ 5388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49521: BlockingRecv error: recv got EOF from 127.0.97.3:49521 (error 108)
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":75}
Sep 24 03:10:04 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:04 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:04.678489 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:04.668816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.3:49039)
0924 03:10:04.669163 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:04.669179 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:04.669274 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:04.669554 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:04.669557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:04.669822 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:04.669966 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:04.669973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.671476 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.671483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:04.673009 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:04.673023 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.673975 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.673981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:04.674172 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:04.674845 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:04.674872 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:04.678167 (+ 3295us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 03:10:04.678670 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:04.668955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49039 (local address 127.0.97.62:39463)
0924 03:10:04.669263 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:04.669268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:04.669289 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:10:04.669359 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:04.669363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:04.669432 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:04.669526 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:04.670283 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.671309 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.673135 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.673794 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.673925 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:04.678383 (+ 4458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49039: BlockingRecv error: recv got EOF from 127.0.97.3:49039 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":210}
W20260924 03:10:04.678848 9636 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 03:10:04 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:04 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:04.681852 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:04.670657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.1:38781)
0924 03:10:04.671264 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:04.671279 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:04.671405 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:04.671696 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:04.671701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:04.672307 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:04.672564 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:04.672580 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.674021 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.674027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:04.675646 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:04.675664 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.676820 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.676827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:04.676977 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:04.677667 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:04.677685 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:04.681633 (+ 3948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":276,"threads_started":1}
W20260924 03:10:04.682188 9640 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:04.670740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38781 (local address 127.0.97.62:39463)
0924 03:10:04.670920 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:04.670932 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:04.671358 (+ 426us) server_negotiation.cc:407] Connection header received
0924 03:10:04.671514 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:04.671518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:04.671570 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:04.671639 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:04.672696 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.673794 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.675834 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.676534 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.676708 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:04.681826 (+ 5118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38781: BlockingRecv error: recv got EOF from 127.0.97.1:38781 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 03:10:05 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:05 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:05.187597 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:05.178216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.2:40099)
0924 03:10:05.178761 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:05.178781 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:05.178959 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:05.179467 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:05.179472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:05.180080 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:05.180276 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:05.180285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:05.181614 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:05.181618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:05.182748 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:05.182760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:05.183547 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:05.183589 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:05.183756 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:05.184341 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:05.184357 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:05.187324 (+ 2967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":305,"threads_started":1}
W20260924 03:10:05.187829 9640 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:05.178396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40099 (local address 127.0.97.62:39463)
0924 03:10:05.178610 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:05.178614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:05.178903 (+ 289us) server_negotiation.cc:407] Connection header received
0924 03:10:05.179103 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:05.179118 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:05.179205 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:05.179330 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:05.180426 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:05.181417 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:05.182892 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:05.183381 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:05.183484 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:05.187502 (+ 4018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40099: BlockingRecv error: recv got EOF from 127.0.97.2:40099 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 24 03:10:05 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:05 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:05.689692 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:05.679442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.3:57485)
0924 03:10:05.680136 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:05.680157 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:05.680285 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:05.680659 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:05.680662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:05.681195 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:05.681388 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:05.681396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:05.682711 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:05.682717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:05.684150 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:05.684162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:05.685164 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:05.685169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:05.685309 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:05.685984 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:05.686003 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:05.689331 (+ 3328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":291,"threads_started":1}
W20260924 03:10:05.689873 9640 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:05.679533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57485 (local address 127.0.97.62:39463)
0924 03:10:05.679772 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:05.679778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:05.680217 (+ 439us) server_negotiation.cc:407] Connection header received
0924 03:10:05.680439 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:05.680442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:05.680529 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:05.680609 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:05.681596 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:05.682570 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:05.684281 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:05.684937 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:05.685065 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:05.689584 (+ 4519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57485: BlockingRecv error: recv got EOF from 127.0.97.3:57485 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 24 03:10:05 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:05 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:05.692330 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:05.682766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.1:56841)
0924 03:10:05.683501 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:05.683517 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:05.683618 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:05.683933 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:05.683936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:05.684308 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:05.684553 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:05.684570 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:05.685774 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:05.685782 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:05.686838 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:05.686851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:05.687812 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:05.687816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:05.688007 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:05.688786 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:05.688807 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:05.691864 (+ 3057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":502,"threads_started":1}
W20260924 03:10:05.692428 9645 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:05.682805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56841 (local address 127.0.97.62:39463)
0924 03:10:05.683315 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:05.683322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:05.683600 (+ 278us) server_negotiation.cc:407] Connection header received
0924 03:10:05.683712 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:05.683717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:05.683786 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:05.683887 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:05.684690 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:05.685641 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:05.687005 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:05.687663 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:05.687795 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:05.692156 (+ 4361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56841: BlockingRecv error: recv got EOF from 127.0.97.1:56841 (error 108)
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":249,"threads_started":1}
Sep 24 03:10:06 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:06 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:06.200835 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:06.188545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.2:35147)
0924 03:10:06.189138 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:06.189153 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:06.189265 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:06.189587 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:06.189590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:06.189965 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:06.190213 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:06.190231 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:06.191304 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:06.191308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:06.192638 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:06.192653 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:06.194110 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:06.194117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:06.194368 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:06.195441 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:06.195471 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:06.200444 (+ 4973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":309,"threads_started":1}
W20260924 03:10:06.200928 9645 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:06.188622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35147 (local address 127.0.97.62:39463)
0924 03:10:06.188785 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:06.188790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:06.189237 (+ 447us) server_negotiation.cc:407] Connection header received
0924 03:10:06.189370 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:06.189374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:06.189444 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:06.189539 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:06.190382 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:06.191158 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:06.192850 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:06.193866 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:06.194020 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:06.200686 (+ 6666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35147: BlockingRecv error: recv got EOF from 127.0.97.2:35147 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 03:10:06 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:06 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:06.702551 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:06.690921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.3:40155)
0924 03:10:06.691472 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:06.691494 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:06.691672 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:06.692098 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:06.692104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:06.692734 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:06.692986 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:06.693000 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:06.694784 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:06.694790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:06.696523 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:06.696539 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:06.697587 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:06.697594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:06.697777 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:06.698457 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:06.698498 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:06.702176 (+ 3678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":371,"threads_started":1}
W20260924 03:10:06.702651 9645 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:06.691085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40155 (local address 127.0.97.62:39463)
0924 03:10:06.691294 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:06.691300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:06.691568 (+ 268us) server_negotiation.cc:407] Connection header received
0924 03:10:06.691831 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:06.691837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:06.691917 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:06.692030 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:06.693189 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:06.694597 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:06.696699 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:06.697429 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:06.697556 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:06.702363 (+ 4807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40155: BlockingRecv error: recv got EOF from 127.0.97.3:40155 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 03:10:06 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:06 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:06.706562 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:06.693327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.1:57217)
0924 03:10:06.693794 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:06.693814 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:06.693920 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:06.694986 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:06.694991 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:06.695476 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:06.695684 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:06.695697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:06.697354 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:06.697359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:06.698994 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:06.699008 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:06.700359 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:06.700365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:06.700572 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:06.703194 (+ 2622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:06.703216 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:06.706253 (+ 3037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":249,"threads_started":1}
W20260924 03:10:06.706816 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:06.693984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57217 (local address 127.0.97.62:39463)
0924 03:10:06.694670 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:06.694677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:06.694696 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:10:06.694744 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:06.694748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:06.694825 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:06.694935 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:06.695854 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:06.697189 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:06.699145 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:06.699886 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:06.699981 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:06.706544 (+ 6563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57217: BlockingRecv error: recv got EOF from 127.0.97.1:57217 (error 108)
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":385,"threads_started":1}
Sep 24 03:10:07 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:07 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:07.211117 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:07.202152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.2:38151)
0924 03:10:07.202692 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:07.202714 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:07.202873 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:07.203423 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:07.203430 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:07.203886 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:07.204227 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:07.204239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:07.205427 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:07.205431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:07.206456 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:07.206467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:07.207207 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:07.207211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:07.207350 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:07.207939 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:07.207966 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:07.210847 (+ 2881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":283,"threads_started":1}
W20260924 03:10:07.211290 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:07.202266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38151 (local address 127.0.97.62:39463)
0924 03:10:07.202496 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:07.202504 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:07.202807 (+ 303us) server_negotiation.cc:407] Connection header received
0924 03:10:07.203168 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:07.203173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:07.203254 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:07.203364 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:07.204395 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:07.205308 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:07.206637 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:07.207053 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:07.207155 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:07.211072 (+ 3917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38151: BlockingRecv error: recv got EOF from 127.0.97.2:38151 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 03:10:07 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:07 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:07.712990 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:07.703496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.3:52539)
0924 03:10:07.704013 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:07.704032 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:07.704153 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:07.704567 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:07.704573 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:07.704920 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:07.705083 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:07.705091 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:07.706228 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:07.706233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:07.707132 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:07.707143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:07.708124 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:07.708130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:07.708343 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:07.709107 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:07.709129 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:07.712692 (+ 3563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":292,"threads_started":1}
W20260924 03:10:07.713176 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:07.703638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52539 (local address 127.0.97.62:39463)
0924 03:10:07.703833 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:07.703838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:07.704158 (+ 320us) server_negotiation.cc:407] Connection header received
0924 03:10:07.704283 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:07.704287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:07.704366 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:07.704458 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:07.705282 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:07.706049 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:07.707287 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:07.707955 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:07.708092 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:07.712875 (+ 4783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52539: BlockingRecv error: recv got EOF from 127.0.97.3:52539 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 03:10:07 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:07 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:07.717386 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:07.707719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.1:54121)
0924 03:10:07.708234 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:07.708259 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:07.708356 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:07.708873 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:07.708880 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:07.709328 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:07.709552 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:07.709568 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:07.711150 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:07.711156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:07.712611 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:07.712626 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:07.713634 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:07.713639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:07.713771 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:07.714356 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:07.714373 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:07.717161 (+ 2788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":263,"threads_started":1}
W20260924 03:10:07.717648 9653 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:07.707811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54121 (local address 127.0.97.62:39463)
0924 03:10:07.708537 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:07.708543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:07.708561 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:10:07.708609 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:07.708615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:07.708695 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:07.708793 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:07.709693 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:07.710924 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:07.712796 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:07.713484 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:07.713587 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:07.717333 (+ 3746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54121: BlockingRecv error: recv got EOF from 127.0.97.1:54121 (error 108)
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":779,"threads_started":1}
Sep 24 03:10:08 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:08 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:08.221192 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:08.212207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.2:52527)
0924 03:10:08.212891 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:08.212906 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:08.213022 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:08.213455 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:08.213462 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:08.213987 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:08.214215 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:08.214230 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:08.215262 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:08.215266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:08.216312 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:08.216324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:08.217240 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:08.217245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:08.217376 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:08.218155 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:08.218173 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:08.220969 (+ 2796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":454,"threads_started":1}
W20260924 03:10:08.221416 9653 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:08.212341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52527 (local address 127.0.97.62:39463)
0924 03:10:08.212605 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:08.212610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:08.213006 (+ 396us) server_negotiation.cc:407] Connection header received
0924 03:10:08.213142 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:08.213146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:08.213209 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:08.213357 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:08.214382 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:08.215107 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:08.216445 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:08.216974 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:08.217206 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:08.221208 (+ 4002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52527: BlockingRecv error: recv got EOF from 127.0.97.2:52527 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 24 03:10:08 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:08 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:08 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:08 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:08.729789 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:08.714102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.3:42971)
0924 03:10:08.714628 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:08.714650 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:08.714764 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:08.715077 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:08.715081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:08.715448 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:08.715644 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:08.715653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:08.716733 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:08.716737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:08.717683 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:08.717694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:08.718429 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:08.718435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:08.718688 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:08.725652 (+ 6964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:08.725680 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:08.729514 (+ 3834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":278,"threads_started":1}
W20260924 03:10:08.730094 9653 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:08.714200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42971 (local address 127.0.97.62:39463)
0924 03:10:08.714381 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:08.714385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:08.714723 (+ 338us) server_negotiation.cc:407] Connection header received
0924 03:10:08.714857 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:08.714860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:08.714918 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:08.715000 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:08.715780 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:08.716601 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:08.717822 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:08.718287 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:08.718413 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:08.729757 (+ 11344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42971: BlockingRecv error: recv got EOF from 127.0.97.3:42971 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 03:10:08.730139 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:08.718332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.1:57097)
0924 03:10:08.719005 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:08.719025 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:08.719184 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:08.719527 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:08.719530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:08.719868 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:08.720014 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:08.720022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:08.721119 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:08.721127 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:08.722271 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:08.722283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:08.722945 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:08.722950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:08.723128 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:08.725652 (+ 2524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:08.725680 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:08.729815 (+ 4135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":302,"threads_started":1}
W20260924 03:10:08.730409 9657 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:08.718401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57097 (local address 127.0.97.62:39463)
0924 03:10:08.719211 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:08.719219 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:08.719238 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:10:08.719290 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:08.719295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:08.719367 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:08.719456 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:08.720134 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:08.720996 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:08.722406 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:08.722804 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:08.722921 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:08.730115 (+ 7194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57097: BlockingRecv error: recv got EOF from 127.0.97.1:57097 (error 108)
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":530,"threads_started":1}
Sep 24 03:10:09 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:09 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:09.230875 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:09.222394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.2:53117)
0924 03:10:09.222904 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:09.222920 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:09.223068 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:09.223481 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:09.223485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:09.223849 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:09.224001 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:09.224009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:09.225052 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:09.225056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:09.226003 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:09.226013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:09.226671 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:09.226675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:09.226785 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:09.227349 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:09.227366 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:09.230512 (+ 3146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":373,"threads_started":1}
W20260924 03:10:09.230988 9657 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:09.222568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53117 (local address 127.0.97.62:39463)
0924 03:10:09.222759 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:09.222764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:09.223006 (+ 242us) server_negotiation.cc:407] Connection header received
0924 03:10:09.223264 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:09.223269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:09.223346 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:09.223448 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:09.224141 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:09.224927 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:09.226141 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:09.226546 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:09.226648 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:09.230755 (+ 4107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53117: BlockingRecv error: recv got EOF from 127.0.97.2:53117 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 24 03:10:09 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:09 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:09 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:09 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 03:10:09.752624 9662 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:09.731496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53577 (local address 127.0.97.62:39463)
0924 03:10:09.741838 (+ 10342us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:09.741845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:09.742248 (+ 403us) server_negotiation.cc:407] Connection header received
0924 03:10:09.742402 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:09.742409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:09.742504 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:09.742609 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:09.743770 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:09.744863 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:09.746460 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:09.747149 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:09.747262 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:09.752264 (+ 5002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53577: BlockingRecv error: recv got EOF from 127.0.97.3:53577 (error 108)
Metrics: {"server-negotiator.queue_time_us":10197,"thread_start_us":10654,"threads_started":1}
I20260924 03:10:09.752696 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:09.731298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.1:49379)
0924 03:10:09.742661 (+ 11363us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:09.742679 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:09.742810 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:09.743362 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:09.743365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:09.743858 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:09.744064 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:09.744077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:09.745732 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:09.745737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:09.746945 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:09.746963 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:09.748188 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:09.748195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:09.748328 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:09.748937 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:09.748955 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:09.752297 (+ 3342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":11252,"thread_start_us":11109,"threads_started":1}
I20260924 03:10:09.752959 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:09.731316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.3:53577)
0924 03:10:09.742063 (+ 10747us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:09.742091 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:09.742242 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:09.742680 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:09.742684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:09.743392 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:09.743595 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:09.743618 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:09.745170 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:09.745176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:09.746304 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:09.746317 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:09.747325 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:09.747331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:09.747534 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:09.748253 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:09.748283 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:09.752011 (+ 3728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":10571,"thread_start_us":10438,"threads_started":1}
W20260924 03:10:09.753213 9661 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:09.732898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49379 (local address 127.0.97.62:39463)
0924 03:10:09.742479 (+ 9581us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:09.742485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:09.742775 (+ 290us) server_negotiation.cc:407] Connection header received
0924 03:10:09.742931 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:09.742938 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:09.743163 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:09.743288 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:09.744247 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:09.745514 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:09.747150 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:09.747947 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:09.748094 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:09.752629 (+ 4535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49379: BlockingRecv error: recv got EOF from 127.0.97.1:49379 (error 108)
Metrics: {"server-negotiator.queue_time_us":9420,"thread_start_us":8623,"threads_started":1}
Sep 24 03:10:10 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:10 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:10.241999 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:10.232096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.2:33697)
0924 03:10:10.232798 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:10.232814 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:10.233000 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:10.233583 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:10.233588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:10.234011 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:10.234197 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:10.234206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:10.235264 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:10.235269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:10.236173 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:10.236184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:10.236984 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:10.236990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:10.237173 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:10.238116 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:10.238145 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:10.241727 (+ 3582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":488,"threads_started":1}
W20260924 03:10:10.242241 9661 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:10.232098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33697 (local address 127.0.97.62:39463)
0924 03:10:10.232466 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:10.232474 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:10.232916 (+ 442us) server_negotiation.cc:407] Connection header received
0924 03:10:10.233252 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:10.233257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:10.233321 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:10.233472 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:10.234352 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:10.235112 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:10.236324 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:10.236817 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:10.236969 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:10.241872 (+ 4903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33697: BlockingRecv error: recv got EOF from 127.0.97.2:33697 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 24 03:10:10 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:10 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:10 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:10 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:10.773360 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:10.754033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.3:49439)
0924 03:10:10.762267 (+ 8234us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:10.762295 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:10.762491 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:10.762973 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:10.762978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:10.763519 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:10.763757 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:10.763769 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:10.765256 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:10.765261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:10.766973 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:10.766992 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:10.768021 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:10.768026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:10.768173 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:10.769179 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:10.769204 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:10.773027 (+ 3823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":8116,"thread_start_us":8018,"threads_started":1}
I20260924 03:10:10.773805 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:10.753780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.1:33233)
0924 03:10:10.762758 (+ 8978us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:10.762774 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:10.762944 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:10.763314 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:10.763318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:10.763838 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:10.764000 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:10.764009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:10.765438 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:10.765443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:10.767253 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:10.767266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:10.768308 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:10.768316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:10.768529 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:10.769179 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:10.769204 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:10.773486 (+ 4282us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":8840,"thread_start_us":8608,"threads_started":1}
W20260924 03:10:10.774252 9667 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:10.754229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49439 (local address 127.0.97.62:39463)
0924 03:10:10.761890 (+ 7661us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:10.761895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:10.762633 (+ 738us) server_negotiation.cc:407] Connection header received
0924 03:10:10.762690 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:10.762696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:10.762778 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:10.762946 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:10.763903 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:10.765096 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:10.767161 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:10.767837 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:10.767972 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:10.773384 (+ 5412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49439: BlockingRecv error: recv got EOF from 127.0.97.3:49439 (error 108)
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":7563,"thread_start_us":7414,"threads_started":1}
W20260924 03:10:10.774609 9665 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:10.753859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33233 (local address 127.0.97.62:39463)
0924 03:10:10.761440 (+ 7581us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:10.761445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:10.762874 (+ 1429us) server_negotiation.cc:407] Connection header received
0924 03:10:10.763043 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:10.763050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:10.763130 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:10.763234 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:10.764142 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:10.765291 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:10.767384 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:10.768137 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:10.768236 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:10.774288 (+ 6052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33233: BlockingRecv error: recv got EOF from 127.0.97.1:33233 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":7465,"thread_start_us":7332,"threads_started":1}
Sep 24 03:10:11 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:11 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:11.252451 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:11.242939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.2:51033)
0924 03:10:11.243679 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:11.243696 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:11.243813 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:11.244164 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:11.244176 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:11.244679 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:11.244882 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:11.244892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:11.246043 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:11.246048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:11.246972 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:11.246983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:11.247853 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:11.247857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:11.247982 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:11.248835 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:11.248854 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:11.252208 (+ 3354us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":404,"threads_started":1}
W20260924 03:10:11.252837 9665 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:11.243112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51033 (local address 127.0.97.62:39463)
0924 03:10:11.243352 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:11.243358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:11.243777 (+ 419us) server_negotiation.cc:407] Connection header received
0924 03:10:11.243925 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:11.243930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:11.244006 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:11.244084 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:11.245022 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:11.245891 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:11.247271 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:11.247700 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:11.247803 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:11.252469 (+ 4666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51033: BlockingRecv error: recv got EOF from 127.0.97.2:51033 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 24 03:10:11 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:11 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:11 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:11 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:11.786368 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:11.774804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.1:51797)
0924 03:10:11.776025 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:11.776044 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:11.776148 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:11.776706 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:11.776713 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:11.777171 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:11.777308 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:11.777317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:11.778751 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:11.778758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:11.780326 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:11.780343 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:11.781522 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:11.781529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:11.781717 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:11.782377 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:11.782399 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:11.786041 (+ 3642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1113,"thread_start_us":1020,"threads_started":1}
I20260924 03:10:11.786618 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:11.774630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.3:59717)
0924 03:10:11.775314 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:11.775358 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:11.775535 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:11.776102 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:11.776108 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:11.776652 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:11.776900 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:11.776917 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:11.778462 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:11.778466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:11.780205 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:11.780223 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:11.781420 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:11.781427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:11.781602 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:11.782360 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:11.782386 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:11.786267 (+ 3881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":231,"threads_started":1}
W20260924 03:10:11.786649 9672 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:11.775468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51797 (local address 127.0.97.62:39463)
0924 03:10:11.776434 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:11.776438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:11.776450 (+ 12us) server_negotiation.cc:407] Connection header received
0924 03:10:11.776481 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:11.776505 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:11.776550 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:11.776635 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:11.777480 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:11.778598 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:11.780456 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:11.781327 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:11.781432 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:11.786320 (+ 4888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51797: BlockingRecv error: recv got EOF from 127.0.97.1:51797 (error 108)
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":889,"thread_start_us":767,"threads_started":1}
W20260924 03:10:11.787088 9669 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:11.774657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59717 (local address 127.0.97.62:39463)
0924 03:10:11.775496 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:11.775502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:11.775524 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:10:11.775699 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:11.775708 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:11.775821 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:11.775968 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:11.777077 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:11.778304 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:11.780347 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:11.781177 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:11.781310 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:11.786509 (+ 5199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59717: BlockingRecv error: recv got EOF from 127.0.97.3:59717 (error 108)
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":733,"thread_start_us":637,"threads_started":1}
Sep 24 03:10:12 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:12 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:12.264312 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:12.253499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.2:39599)
0924 03:10:12.254152 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:12.254170 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:12.254288 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:12.254716 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:12.254719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:12.255284 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:12.255450 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:12.255460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:12.256751 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:12.256756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:12.257871 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:12.257884 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:12.258821 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:12.258827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:12.258962 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:12.259721 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:12.259741 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:12.263961 (+ 4220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":303,"threads_started":1}
W20260924 03:10:12.264390 9669 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:12.253713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39599 (local address 127.0.97.62:39463)
0924 03:10:12.253891 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:12.253896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:12.254290 (+ 394us) server_negotiation.cc:407] Connection header received
0924 03:10:12.254466 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:12.254470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:12.254541 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:12.254630 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:12.255617 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:12.256578 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:12.258215 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:12.258623 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:12.258755 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:12.264171 (+ 5416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39599: BlockingRecv error: recv got EOF from 127.0.97.2:39599 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 03:10:12 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:12 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:12 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:12 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:12.809293 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:12.787467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.1:36383)
0924 03:10:12.799637 (+ 12170us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:12.799660 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:12.799801 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:12.800259 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:12.800266 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:12.800856 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:12.801025 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:12.801034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:12.802257 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:12.802263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:12.803375 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:12.803386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:12.804525 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:12.804531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:12.804725 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:12.805729 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:12.805747 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:12.808937 (+ 3190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":12030,"thread_start_us":11886,"threads_started":1}
W20260924 03:10:12.809453 9677 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:12.787485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36383 (local address 127.0.97.62:39463)
0924 03:10:12.798967 (+ 11482us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:12.798978 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:12.799780 (+ 802us) server_negotiation.cc:407] Connection header received
0924 03:10:12.799965 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:12.799972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:12.800068 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:12.800201 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:12.801179 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:12.802123 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:12.803589 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:12.804297 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:12.804452 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:12.809149 (+ 4697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36383: BlockingRecv error: recv got EOF from 127.0.97.1:36383 (error 108)
Metrics: {"server-negotiator.queue_time_us":11296,"spinlock_wait_cycles":20736,"thread_start_us":11424,"threads_started":1}
I20260924 03:10:12.809695 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:12.787698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.3:50007)
0924 03:10:12.799783 (+ 12085us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:12.799800 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:12.799923 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:12.800661 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:12.800666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:12.801263 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:12.801450 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:12.801464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:12.802834 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:12.802839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:12.804333 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:12.804352 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:12.805361 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:12.805368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:12.805580 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:12.806232 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:12.806251 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:12.809361 (+ 3110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":11979,"thread_start_us":11873,"threads_started":1}
W20260924 03:10:12.810057 9674 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:12.787790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50007 (local address 127.0.97.62:39463)
0924 03:10:12.799219 (+ 11429us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:12.799224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:12.799891 (+ 667us) server_negotiation.cc:407] Connection header received
0924 03:10:12.800203 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:12.800208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:12.800288 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:12.800606 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:12.801632 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:12.802672 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:12.804481 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:12.805145 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:12.805234 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:12.809680 (+ 4446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50007: BlockingRecv error: recv got EOF from 127.0.97.3:50007 (error 108)
Metrics: {"server-negotiator.queue_time_us":11308,"thread_start_us":10825,"threads_started":1}
Sep 24 03:10:13 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:13 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:13.276001 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:13.265465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.2:51895)
0924 03:10:13.266179 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:13.266197 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:13.266380 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:13.266882 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:13.266886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:13.267449 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:13.267651 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:13.267661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:13.268936 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:13.268940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:13.269988 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:13.270000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:13.270899 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:13.270908 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:13.271118 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:13.271898 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:13.271921 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:13.275738 (+ 3817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":340,"threads_started":1}
W20260924 03:10:13.276394 9674 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:13.265483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51895 (local address 127.0.97.62:39463)
0924 03:10:13.265664 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:13.265672 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:13.266301 (+ 629us) server_negotiation.cc:407] Connection header received
0924 03:10:13.266526 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:13.266532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:13.266664 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:13.266785 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:13.267801 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:13.268781 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:13.270134 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:13.270676 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:13.270820 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:13.276142 (+ 5322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51895: BlockingRecv error: recv got EOF from 127.0.97.2:51895 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 03:10:13 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:13 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:13 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:13 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:13.841842 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:13.810518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.1:51945)
0924 03:10:13.823010 (+ 12492us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:13.823034 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:13.823148 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:13.823641 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:13.823646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:13.824115 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:13.824306 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:13.824320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:13.826061 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:13.826067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:13.827671 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:13.827687 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:13.829149 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:13.829165 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:13.829365 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:13.837181 (+ 7816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:13.837218 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:13.841498 (+ 4280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":12369,"thread_start_us":12246,"threads_started":1}
W20260924 03:10:13.842267 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:13.810751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51945 (local address 127.0.97.62:39463)
0924 03:10:13.822683 (+ 11932us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:13.822690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:13.823139 (+ 449us) server_negotiation.cc:407] Connection header received
0924 03:10:13.823231 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:13.823236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:13.823330 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:13.823457 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:13.824593 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:13.825893 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:13.827860 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:13.828809 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:13.828960 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:13.841880 (+ 12920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51945: BlockingRecv error: recv got EOF from 127.0.97.1:51945 (error 108)
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":11824,"spinlock_wait_cycles":28416,"thread_start_us":11726,"threads_started":1}
I20260924 03:10:13.842377 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:13.810869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.3:42423)
0924 03:10:13.822471 (+ 11602us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:13.822508 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:13.822656 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:13.823608 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:13.823613 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:13.824194 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:13.824418 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:13.824434 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:13.826061 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:13.826066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:13.827671 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:13.827687 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:13.829200 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:13.829210 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:13.829390 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:13.837281 (+ 7891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:13.837310 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:13.841498 (+ 4188us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":11450,"thread_start_us":11311,"threads_started":1}
W20260924 03:10:13.843106 9682 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:13.811252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42423 (local address 127.0.97.62:39463)
0924 03:10:13.823158 (+ 11906us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:13.823165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:13.823180 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:10:13.823231 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:13.823236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:13.823330 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:13.823459 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:13.824629 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:13.825893 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:13.827860 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:13.828809 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:13.829353 (+ 544us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:13.842125 (+ 12772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42423: BlockingRecv error: recv got EOF from 127.0.97.3:42423 (error 108)
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":11783,"spinlock_wait_cycles":11008,"thread_start_us":11595,"threads_started":1}
Sep 24 03:10:14 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:14 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:14.286607 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:14.277057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.2:53109)
0924 03:10:14.277601 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:14.277632 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:14.277770 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:14.278329 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:14.278332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:14.278744 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:14.278907 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:14.278917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:14.280335 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:14.280341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:14.281445 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:14.281458 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:14.282355 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:14.282360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:14.282492 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:14.283100 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:14.283119 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:14.286246 (+ 3127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":275,"threads_started":1}
W20260924 03:10:14.286723 9682 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:14.277233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53109 (local address 127.0.97.62:39463)
0924 03:10:14.277882 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:14.277888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:14.277903 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:10:14.277959 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:14.277963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:14.278018 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:14.278127 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:14.279217 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:14.280074 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:14.281610 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:14.282133 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:14.282301 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:14.286477 (+ 4176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53109: BlockingRecv error: recv got EOF from 127.0.97.2:53109 (error 108)
Metrics: {"server-negotiator.queue_time_us":510}
Sep 24 03:10:14 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:14 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:14 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:14.854703 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:14.843438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.1:52625)
0924 03:10:14.844088 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:14.844110 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:14.844261 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:14.844951 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:14.844956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:14.845523 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:14.845709 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:14.845718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:14.846925 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:14.846933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:14.848275 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:14.848298 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:14.849453 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:14.849459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:14.849708 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:14.850679 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:14.850701 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:14.854215 (+ 3514us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":360,"threads_started":1}
W20260924 03:10:14.854996 9685 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:14.843682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52625 (local address 127.0.97.62:39463)
0924 03:10:14.844599 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:14.844608 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:14.844633 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:10:14.844706 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:14.844711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:14.844801 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:14.844915 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:14.845887 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:14.846787 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:14.848478 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:14.849270 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:14.849413 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:14.854537 (+ 5124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52625: BlockingRecv error: recv got EOF from 127.0.97.1:52625 (error 108)
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":357,"threads_started":1}
Sep 24 03:10:14 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:14.856297 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:14.843949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.3:38015)
0924 03:10:14.844996 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:14.845014 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:14.845197 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:14.845605 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:14.845611 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:14.846002 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:14.846174 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:14.846183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:14.847871 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:14.847876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:14.849761 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:14.849793 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:14.851076 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:14.851084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:14.851285 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:14.851932 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:14.851961 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:14.856044 (+ 4083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":878,"thread_start_us":703,"threads_started":1}
W20260924 03:10:14.856866 9687 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:14.844295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38015 (local address 127.0.97.62:39463)
0924 03:10:14.844827 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:14.844832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:14.845125 (+ 293us) server_negotiation.cc:407] Connection header received
0924 03:10:14.845350 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:14.845355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:14.845441 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:14.845575 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:14.846390 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:14.847662 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:14.850087 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:14.850849 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:14.851031 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:14.856613 (+ 5582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38015: BlockingRecv error: recv got EOF from 127.0.97.3:38015 (error 108)
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":269,"threads_started":1}
Sep 24 03:10:15 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:15 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:15.297696 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:15.288110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.2:32933)
0924 03:10:15.288654 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:15.288671 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:15.288824 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:15.289189 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:15.289194 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:15.289615 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:15.289809 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:15.289817 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:15.290870 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:15.290876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:15.291781 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:15.291821 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:15.292925 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:15.292932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:15.293100 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:15.293839 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:15.293868 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:15.297436 (+ 3568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":253,"threads_started":1}
W20260924 03:10:15.297887 9687 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:15.288175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32933 (local address 127.0.97.62:39463)
0924 03:10:15.288394 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:15.288402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:15.288755 (+ 353us) server_negotiation.cc:407] Connection header received
0924 03:10:15.288984 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:15.288995 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:15.289059 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:15.289160 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:15.289947 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:15.290697 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:15.291970 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:15.292725 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:15.292829 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:15.297650 (+ 4821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:32933: BlockingRecv error: recv got EOF from 127.0.97.2:32933 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 03:10:15 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:15 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:15 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:15.866645 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:15.855845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.1:49237)
0924 03:10:15.856409 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:15.856426 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:15.856650 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:15.857028 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:15.857032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:15.857476 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:15.857719 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:15.857737 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:15.859033 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:15.859039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:15.860237 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:15.860251 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:15.861305 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:15.861311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:15.861518 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:15.862371 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:15.862395 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:15.866120 (+ 3725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":327,"threads_started":1}
W20260924 03:10:15.866781 9690 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:15.855907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49237 (local address 127.0.97.62:39463)
0924 03:10:15.856738 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:15.856743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:15.856758 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:10:15.856801 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:15.856804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:15.856867 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:15.856954 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:15.857909 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:15.858880 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:15.860447 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:15.861142 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:15.861226 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:15.866335 (+ 5109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49237: BlockingRecv error: recv got EOF from 127.0.97.1:49237 (error 108)
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":668,"threads_started":1}
Sep 24 03:10:15 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:15.868130 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:15.857299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.3:54893)
0924 03:10:15.858223 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:15.858239 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:15.858345 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:15.858700 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:15.858703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:15.859285 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:15.859508 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:15.859520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:15.861072 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:15.861076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:15.862566 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:15.862584 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:15.863536 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:15.863545 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:15.863738 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:15.864442 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:15.864468 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:15.867897 (+ 3429us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":617,"threads_started":1}
W20260924 03:10:15.868642 9692 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:15.857389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54893 (local address 127.0.97.62:39463)
0924 03:10:15.857942 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:15.857947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:15.858327 (+ 380us) server_negotiation.cc:407] Connection header received
0924 03:10:15.858449 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:15.858454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:15.858542 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:15.858653 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:15.859701 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:15.860905 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:15.862739 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:15.863329 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:15.863483 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:15.868339 (+ 4856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54893: BlockingRecv error: recv got EOF from 127.0.97.3:54893 (error 108)
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":315,"threads_started":1}
Sep 24 03:10:16 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:16 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:16.308245 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:16.298792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.2:44547)
0924 03:10:16.299502 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:16.299527 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:16.299646 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:16.299959 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:16.300044 (+ 85us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:16.300427 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:16.300665 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:16.300674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:16.301904 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:16.301909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:16.303181 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:16.303193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:16.304006 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:16.304011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:16.304204 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:16.304899 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:16.304916 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:16.307996 (+ 3080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":467,"threads_started":1}
W20260924 03:10:16.308525 9692 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:16.298926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44547 (local address 127.0.97.62:39463)
0924 03:10:16.299094 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:16.299099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:16.299613 (+ 514us) server_negotiation.cc:407] Connection header received
0924 03:10:16.299751 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:16.299754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:16.299816 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:16.299907 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:16.300811 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:16.301739 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:16.303309 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:16.303879 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:16.303973 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:16.308253 (+ 4280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44547: BlockingRecv error: recv got EOF from 127.0.97.2:44547 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 03:10:16 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:16 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:16 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:16 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:16.892884 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:16.867881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.1:55871)
0924 03:10:16.881998 (+ 14117us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:16.882028 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:16.882217 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:16.882823 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:16.882829 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:16.883437 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:16.883708 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:16.883723 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:16.885147 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:16.885155 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:16.886294 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:16.886307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:16.887420 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:16.887431 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:16.887675 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:16.888539 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:16.888570 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:16.892526 (+ 3956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":13997,"thread_start_us":13836,"threads_started":1}
W20260924 03:10:16.893286 9695 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:16.867888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55871 (local address 127.0.97.62:39463)
0924 03:10:16.881432 (+ 13544us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:16.881441 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:16.882328 (+ 887us) server_negotiation.cc:407] Connection header received
0924 03:10:16.882387 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:16.882393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:16.882477 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:16.882580 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:16.883873 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:16.885010 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:16.886484 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:16.887138 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:16.887262 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:16.892740 (+ 5478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55871: BlockingRecv error: recv got EOF from 127.0.97.1:55871 (error 108)
Metrics: {"server-negotiator.queue_time_us":13361,"thread_start_us":13570,"threads_started":1}
I20260924 03:10:16.894143 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:16.869070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.3:41173)
0924 03:10:16.883456 (+ 14386us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:16.883476 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:16.883622 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:16.884058 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:16.884063 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:16.884536 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:16.884716 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:16.884727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:16.886419 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:16.886424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:16.887842 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:16.887854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:16.888906 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:16.888912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:16.889120 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:16.889696 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:16.889714 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:16.893699 (+ 3985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":14243,"thread_start_us":14120,"threads_started":1}
W20260924 03:10:16.894244 9697 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:16.869470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41173 (local address 127.0.97.62:39463)
0924 03:10:16.883043 (+ 13573us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:16.883051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:16.883598 (+ 547us) server_negotiation.cc:407] Connection header received
0924 03:10:16.883746 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:16.883754 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:16.883863 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:16.884021 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:16.884892 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:16.886264 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:16.887984 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:16.888764 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:16.888887 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:16.893929 (+ 5042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41173: BlockingRecv error: recv got EOF from 127.0.97.3:41173 (error 108)
Metrics: {"server-negotiator.queue_time_us":13441,"thread_start_us":13327,"threads_started":1}
Sep 24 03:10:17 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:17 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:17.318591 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:17.309324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.2:44365)
0924 03:10:17.310197 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:17.310216 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:17.310429 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:17.310814 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:17.310818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:17.311285 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:17.311474 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:17.311483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:17.312567 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:17.312573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:17.313498 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:17.313510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:17.314257 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:17.314263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:17.314439 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:17.315082 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:17.315100 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:17.318210 (+ 3110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":530,"threads_started":1}
W20260924 03:10:17.318625 9697 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:17.309647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44365 (local address 127.0.97.62:39463)
0924 03:10:17.309901 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:17.309909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:17.310341 (+ 432us) server_negotiation.cc:407] Connection header received
0924 03:10:17.310609 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:17.310621 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:17.310684 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:17.310791 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:17.311633 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:17.312374 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:17.313648 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:17.314073 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:17.314217 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:17.318388 (+ 4171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44365: BlockingRecv error: recv got EOF from 127.0.97.2:44365 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 24 03:10:17 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:17 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:17 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:17.904330 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:17.893853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.1:57977)
0924 03:10:17.894441 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:17.894459 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:17.894568 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:17.895143 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:17.895147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:17.895557 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:17.895815 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:17.895830 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:17.897486 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:17.897495 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:17.898835 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:17.898852 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:17.899734 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:17.899740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:17.899923 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:17.900657 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:17.900685 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:17.904001 (+ 3316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":300,"threads_started":1}
W20260924 03:10:17.904690 9700 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:17.894032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57977 (local address 127.0.97.62:39463)
0924 03:10:17.894820 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:17.894824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:17.894840 (+ 16us) server_negotiation.cc:407] Connection header received
0924 03:10:17.894886 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:17.894890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:17.894953 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:17.895060 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:17.895991 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:17.897312 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:17.898992 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:17.899498 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:17.899724 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:17.904420 (+ 4696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57977: BlockingRecv error: recv got EOF from 127.0.97.1:57977 (error 108)
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":554,"threads_started":1}
Sep 24 03:10:17 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:17.906071 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:17.895852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.3:46491)
0924 03:10:17.896584 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:17.896608 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:17.896874 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:17.897180 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:17.897187 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:17.897790 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:17.897969 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:17.898006 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:17.899127 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:17.899135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:17.900061 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:17.900071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:17.900958 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:17.900966 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:17.901130 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:17.901767 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:17.901788 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:17.905752 (+ 3964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":426,"threads_started":1}
W20260924 03:10:17.906217 9702 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:17.895958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46491 (local address 127.0.97.62:39463)
0924 03:10:17.896911 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:17.896919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:17.896938 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:10:17.896984 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:17.896989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:17.897047 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:17.897153 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:17.898145 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:17.898989 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:17.900204 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:17.900774 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:17.900883 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:17.905956 (+ 5073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46491: BlockingRecv error: recv got EOF from 127.0.97.3:46491 (error 108)
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":616,"threads_started":1}
Sep 24 03:10:18 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:18 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:18.328358 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:18.319749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.2:43703)
0924 03:10:18.320324 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:18.320340 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:18.320560 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:18.320952 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:18.320956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:18.321334 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:18.321481 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:18.321489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:18.322539 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:18.322545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:18.323463 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:18.323473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:18.324195 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:18.324200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:18.324408 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:18.325121 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:18.325140 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:18.328032 (+ 2892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":298,"threads_started":1}
W20260924 03:10:18.328642 9702 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:18.319936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43703 (local address 127.0.97.62:39463)
0924 03:10:18.320236 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:18.320243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:18.320434 (+ 191us) server_negotiation.cc:407] Connection header received
0924 03:10:18.320731 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:18.320737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:18.320817 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:18.320926 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:18.321632 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:18.322383 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:18.323607 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:18.324040 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:18.324181 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:18.328371 (+ 4190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43703: BlockingRecv error: recv got EOF from 127.0.97.2:43703 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 03:10:18 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:18 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:18 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:18 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:18.921453 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:18.907272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.3:60379)
0924 03:10:18.907813 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:18.907838 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:18.907970 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:18.908427 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:18.908435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:18.908893 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:18.909103 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:18.909115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:18.910720 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:18.910725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:18.912421 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:18.912438 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:18.913355 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:18.913359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:18.913487 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:18.917340 (+ 3853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:18.917367 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:18.921023 (+ 3656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":248,"threads_started":1}
W20260924 03:10:18.921716 9707 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:18.907320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60379 (local address 127.0.97.62:39463)
0924 03:10:18.908149 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:18.908157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:18.908172 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:10:18.908213 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:18.908216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:18.908278 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:18.908362 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:18.909352 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:18.910551 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:18.912606 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:18.913099 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:18.913346 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:18.921382 (+ 8036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60379: BlockingRecv error: recv got EOF from 127.0.97.3:60379 (error 108)
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":535,"threads_started":1}
I20260924 03:10:18.922067 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:18.905721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.1:37041)
0924 03:10:18.906304 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:18.906322 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:18.906423 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:18.906922 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:18.906928 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:18.907595 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:18.907900 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:18.907917 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:18.909407 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:18.909416 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:18.910862 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:18.910875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:18.911960 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:18.911966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:18.912100 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:18.917340 (+ 5240us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:18.917367 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:18.921691 (+ 4324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":237,"threads_started":1}
W20260924 03:10:18.922399 9705 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:18.905910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37041 (local address 127.0.97.62:39463)
0924 03:10:18.906524 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:18.906553 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:18.906575 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:10:18.906628 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:18.906632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:18.906715 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:18.906835 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:18.908088 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:18.909246 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:18.911024 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:18.911786 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:18.911938 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:18.922171 (+ 10233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37041: BlockingRecv error: recv got EOF from 127.0.97.1:37041 (error 108)
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":384,"threads_started":1}
Sep 24 03:10:19 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:10:19 dist-test-slave-5l98 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:10:19.338713 9708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:19.329689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39463 (local address 127.0.97.2:48357)
0924 03:10:19.330329 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:19.330352 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:19.330514 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:19.330880 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:19.330884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:19.331299 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:19.331470 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:19.331479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:19.332445 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:19.332450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:19.333431 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:19.333442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:19.334170 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:19.334176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:19.334345 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:19.334974 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:19.334992 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:19.338416 (+ 3424us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39463: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":416,"threads_started":1}
W20260924 03:10:19.338930 9705 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:19.329822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48357 (local address 127.0.97.62:39463)
0924 03:10:19.330063 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:19.330072 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:19.330445 (+ 373us) server_negotiation.cc:407] Connection header received
0924 03:10:19.330669 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:19.330674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:19.330735 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:19.330836 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:19.331606 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:19.332312 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:19.333567 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:19.333980 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:19.334149 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:19.338695 (+ 4546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48357: BlockingRecv error: recv got EOF from 127.0.97.2:48357 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 03:10:19.655292 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 9225
I20260924 03:10:19.668605 9710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:19.656459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.0.97.1:40573)
0924 03:10:19.657158 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:19.657184 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:19.657217 (+ 33us) server_negotiation.cc:407] Connection header received
0924 03:10:19.657303 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:19.657314 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:19.657595 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:19.657754 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:19.659344 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:19.660714 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:19.662086 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:19.662631 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:19.662734 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:19.665219 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:19.665229 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:19.665243 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:19.665283 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:19.667925 (+ 2642us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 03:10:19.668279 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43304: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":340,"threads_started":1}
W20260924 03:10:19.668634 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 9225: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:40573: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 03:10:19.668723 9710 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43304: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 03:10:19.668860 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 9225
I20260924 03:10:19.688143 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 9377
I20260924 03:10:19.701612 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:19.689413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42662 (local address 127.0.97.2:40277)
0924 03:10:19.689984 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:19.689991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:19.690072 (+ 81us) server_negotiation.cc:407] Connection header received
0924 03:10:19.690143 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:19.690150 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:19.690364 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:19.690517 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:19.691924 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:19.693304 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:19.694514 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:19.695164 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:19.695273 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:19.698120 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:19.698133 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:19.698148 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:19.698189 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:19.701075 (+ 2886us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 03:10:19.701337 (+ 262us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42662: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":289,"threads_started":1}
W20260924 03:10:19.701762 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 9377: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:40277: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 03:10:19.701862 9711 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42662: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 03:10:19.701931 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 9377
I20260924 03:10:19.715929 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 9508
I20260924 03:10:19.729174 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:19.717004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60250 (local address 127.0.97.3:38195)
0924 03:10:19.717565 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:19.717573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:19.717600 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:10:19.717664 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:19.717670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:19.717893 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:19.718049 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:19.719449 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:19.720666 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:19.722094 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:19.722729 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:19.722835 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:19.725939 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:19.725948 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:19.725962 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:19.726008 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:19.728613 (+ 2605us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 03:10:19.728880 (+ 267us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60250: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":293,"threads_started":1}
W20260924 03:10:19.729283 9712 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60250: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 03:10:19.729525 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 9508: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:38195: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 03:10:19.729704 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 9508
I20260924 03:10:19.747797 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 9156
I20260924 03:10:19.749318 9218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:10:19.868537 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 9156
2026-09-24T03:10:19Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21505 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T03:10:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:10:19Z Disabled control of system clock
I20260924 03:10:19.947557 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45657
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 03:10:20.296123 9717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:20.296413 9717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:20.296540 9717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:20.305105 9717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:10:20.305219 9717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:20.305256 9717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:20.305306 9717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:10:20.305349 9717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:10:20.315451 9717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45657
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:10:20.316792 9717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:20.318559 9717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:10:20.331660 9717 server_base.cc:1034] running on GCE node
W20260924 03:10:20.331707 9722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:20.331845 9725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:20.331609 9723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:20.333110 9717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:20.334587 9717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:20.335877 9717 hybrid_clock.cc:648] HybridClock initialized: now 1790219420335821 us; error 41 us; skew 500 ppm
I20260924 03:10:20.336292 9717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:10:20.339326 9717 webserver.cc:460] Webserver started at http://127.0.97.62:43849/ using document root <none> and password file <none>
I20260924 03:10:20.340107 9717 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:20.340343 9717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:20.340754 9717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:20.343745 9717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "b18064c0c7ed40f7b3bfe9a15a914ad8"
format_stamp: "Formatted at 2026-09-24 03:10:20 on dist-test-slave-5l98"
server_key: "378a3d51652177a567517de0186257f6"
server_key_iv: "ebafe71af1382ebf774b2295f2f82a4e"
server_key_version: "encryptionkey@0"
I20260924 03:10:20.344843 9717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "b18064c0c7ed40f7b3bfe9a15a914ad8"
format_stamp: "Formatted at 2026-09-24 03:10:20 on dist-test-slave-5l98"
server_key: "378a3d51652177a567517de0186257f6"
server_key_iv: "ebafe71af1382ebf774b2295f2f82a4e"
server_key_version: "encryptionkey@0"
I20260924 03:10:20.350648 9717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260924 03:10:20.354539 9731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:20.356065 9717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:10:20.356287 9717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "b18064c0c7ed40f7b3bfe9a15a914ad8"
format_stamp: "Formatted at 2026-09-24 03:10:20 on dist-test-slave-5l98"
server_key: "378a3d51652177a567517de0186257f6"
server_key_iv: "ebafe71af1382ebf774b2295f2f82a4e"
server_key_version: "encryptionkey@0"
I20260924 03:10:20.356587 9717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:20.385390 9717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:20.386389 9717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:20.386744 9717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:20.410778 9717 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45657
I20260924 03:10:20.410799 9782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45657 every 8 connection(s)
I20260924 03:10:20.413009 9717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:10:20.419827 9783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:20.421082 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 9717
I20260924 03:10:20.421375 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:10:20.422017 388 external_mini_cluster.cc:1428] Setting key 1da0177b4f0b5d8f4d7b57ca32487ddc
I20260924 03:10:20.436933 9783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8: Bootstrap starting.
I20260924 03:10:20.440007 9786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:20.424433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57912 (local address 127.0.97.62:45657)
0924 03:10:20.425742 (+ 1309us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:20.425762 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:20.425896 (+ 134us) server_negotiation.cc:407] Connection header received
0924 03:10:20.427987 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:20.428021 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:20.428784 (+ 763us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:20.429299 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:20.430025 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:20.432133 (+ 2108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:20.433999 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:20.435199 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:20.435412 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:20.436937 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:20.436948 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:20.436970 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:10:20.437746 (+ 776us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:20.437825 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:10:20.437852 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:20.437976 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:20.438313 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 03:10:20.438672 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":373,"threads_started":1}
I20260924 03:10:20.441540 9783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:20.442678 9783 log.cc:824] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:20.446013 9783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8: No bootstrap required, opened a new log
I20260924 03:10:20.455498 9783 raft_consensus.cc:348] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18064c0c7ed40f7b3bfe9a15a914ad8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45657 } }
I20260924 03:10:20.455955 9783 raft_consensus.cc:374] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:20.456074 9783 raft_consensus.cc:729] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b18064c0c7ed40f7b3bfe9a15a914ad8, State: Initialized, Role: FOLLOWER
I20260924 03:10:20.457499 9783 consensus_queue.cc:260] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18064c0c7ed40f7b3bfe9a15a914ad8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45657 } }
I20260924 03:10:20.457816 9783 raft_consensus.cc:388] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:10:20.457978 9783 raft_consensus.cc:482] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:10:20.458173 9783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:20.460528 9783 raft_consensus.cc:504] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18064c0c7ed40f7b3bfe9a15a914ad8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45657 } }
I20260924 03:10:20.461061 9783 leader_election.cc:302] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b18064c0c7ed40f7b3bfe9a15a914ad8; no voters:
I20260924 03:10:20.461720 9783 leader_election.cc:288] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:10:20.461930 9788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:10:20.462651 9788 raft_consensus.cc:686] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8 [term 1 LEADER]: Becoming Leader. State: Replica: b18064c0c7ed40f7b3bfe9a15a914ad8, State: Running, Role: LEADER
I20260924 03:10:20.463255 9788 consensus_queue.cc:237] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18064c0c7ed40f7b3bfe9a15a914ad8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45657 } }
I20260924 03:10:20.463967 9783 sys_catalog.cc:564] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:10:20.466066 9790 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b18064c0c7ed40f7b3bfe9a15a914ad8. Latest consensus state: current_term: 1 leader_uuid: "b18064c0c7ed40f7b3bfe9a15a914ad8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18064c0c7ed40f7b3bfe9a15a914ad8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45657 } } }
I20260924 03:10:20.466472 9790 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8 [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:20.467276 9789 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b18064c0c7ed40f7b3bfe9a15a914ad8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18064c0c7ed40f7b3bfe9a15a914ad8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45657 } } }
I20260924 03:10:20.467886 9789 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8 [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:20.468740 9795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:10:20.474824 9795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:10:20.485255 9795 catalog_manager.cc:1347] Generated new cluster ID: 7c3e6a0e71ea4368b0119e4587137465
I20260924 03:10:20.485399 9795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:10:20.500530 9795 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:10:20.501878 9795 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:10:20.512621 9795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b18064c0c7ed40f7b3bfe9a15a914ad8: Generated new TSK 0
I20260924 03:10:20.513739 9795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:10:20.520169 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45657
--builtin_ntp_servers=127.0.97.20:34807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 03:10:20.863371 9807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:20.863739 9807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:20.863889 9807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:20.872121 9807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:20.872242 9807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:20.872423 9807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:10:20.881911 9807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:10:20.883275 9807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:20.885213 9807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:20.899648 9812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:20.899693 9815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:20.899721 9813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:20.900655 9807 server_base.cc:1034] running on GCE node
I20260924 03:10:20.901716 9807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:20.903261 9807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:20.904752 9807 hybrid_clock.cc:648] HybridClock initialized: now 1790219420904651 us; error 58 us; skew 500 ppm
I20260924 03:10:20.905287 9807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:10:20.909039 9807 webserver.cc:460] Webserver started at http://127.0.97.1:41815/ using document root <none> and password file <none>
I20260924 03:10:20.909884 9807 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:20.910058 9807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:20.910429 9807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:20.913836 9807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "8fdee49752044877925bdaf81015a853"
format_stamp: "Formatted at 2026-09-24 03:10:20 on dist-test-slave-5l98"
server_key: "806796f17639124d2c6e4532035eb4a4"
server_key_iv: "e261800c385c13bdfc3cd860082655c5"
server_key_version: "encryptionkey@0"
I20260924 03:10:20.914788 9807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8fdee49752044877925bdaf81015a853"
format_stamp: "Formatted at 2026-09-24 03:10:20 on dist-test-slave-5l98"
server_key: "806796f17639124d2c6e4532035eb4a4"
server_key_iv: "e261800c385c13bdfc3cd860082655c5"
server_key_version: "encryptionkey@0"
I20260924 03:10:20.921170 9807 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 03:10:20.925227 9821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:20.926849 9807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:10:20.927002 9807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "8fdee49752044877925bdaf81015a853"
format_stamp: "Formatted at 2026-09-24 03:10:20 on dist-test-slave-5l98"
server_key: "806796f17639124d2c6e4532035eb4a4"
server_key_iv: "e261800c385c13bdfc3cd860082655c5"
server_key_version: "encryptionkey@0"
I20260924 03:10:20.927206 9807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:20.952696 9807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:20.953660 9807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:20.953900 9807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:20.954865 9807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:20.956400 9807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:20.956533 9807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:20.956641 9807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:20.956701 9807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:20.993002 9807 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38615
I20260924 03:10:20.993063 9933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38615 every 8 connection(s)
I20260924 03:10:20.994263 9807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:10:20.996919 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 9807
I20260924 03:10:20.997109 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:10:20.997859 388 external_mini_cluster.cc:1428] Setting key aa4dbcdb5c13386706446f1829749e8e
I20260924 03:10:21.000842 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45657
--builtin_ntp_servers=127.0.97.20:34807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 03:10:21.009289 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:20.997068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52389 (local address 127.0.97.62:45657)
0924 03:10:20.997545 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:20.997551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:20.998895 (+ 1344us) server_negotiation.cc:407] Connection header received
0924 03:10:21.000733 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:21.000739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:21.000818 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:21.000942 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:21.002253 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:21.003420 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:21.005710 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:21.006448 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:21.006589 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:21.008348 (+ 1759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:21.008354 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:21.008357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:10:21.008376 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:21.008417 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:10:21.008428 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:21.008609 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:21.008862 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 03:10:21.008960 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":233,"threads_started":1}
I20260924 03:10:21.010818 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:20.997716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45657 (local address 127.0.97.1:52389)
0924 03:10:20.998750 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:20.998808 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:21.000385 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:21.001354 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:21.001363 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:21.001464 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:21.002069 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:21.002086 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:21.003746 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:21.003754 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:21.005526 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:21.005544 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:21.006697 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:21.006703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:21.006920 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:21.007879 (+ 959us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:10:21.007884 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:21.007945 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:10:21.008087 (+ 142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:21.008619 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:21.008633 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:21.008645 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 03:10:21.008843 (+ 198us) client_negotiation.cc:240] Negotiation successful
0924 03:10:21.009348 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":336,"threads_started":1}
I20260924 03:10:21.012884 9934 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45657
I20260924 03:10:21.013221 9934 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:21.014192 9934 heartbeater.cc:511] Master 127.0.97.62:45657 requested a full tablet report, sending...
I20260924 03:10:21.016824 9748 ts_manager.cc:194] Registered new tserver with Master: 8fdee49752044877925bdaf81015a853 (127.0.97.1:38615)
I20260924 03:10:21.018369 9748 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:52389
W20260924 03:10:21.339262 9938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:21.339586 9938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:21.339762 9938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:21.348628 9938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:21.348766 9938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:21.349005 9938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:10:21.359200 9938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:10:21.360755 9938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:21.362728 9938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:21.378515 9946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:21.378719 9944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:21.378803 9943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:21.379329 9938 server_base.cc:1034] running on GCE node
I20260924 03:10:21.380014 9938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:21.381218 9938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:21.382506 9938 hybrid_clock.cc:648] HybridClock initialized: now 1790219421382442 us; error 55 us; skew 500 ppm
I20260924 03:10:21.382925 9938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:10:21.385656 9938 webserver.cc:460] Webserver started at http://127.0.97.2:38347/ using document root <none> and password file <none>
I20260924 03:10:21.386394 9938 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:21.386536 9938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:21.386865 9938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:21.389925 9938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "89ec5d0fec174b1686042fc999955d19"
format_stamp: "Formatted at 2026-09-24 03:10:21 on dist-test-slave-5l98"
server_key: "e3100fd718cb8dcbdcc16cc42ed0cd03"
server_key_iv: "9e03d9441984fcbab77e22e382b9bea2"
server_key_version: "encryptionkey@0"
I20260924 03:10:21.390779 9938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "89ec5d0fec174b1686042fc999955d19"
format_stamp: "Formatted at 2026-09-24 03:10:21 on dist-test-slave-5l98"
server_key: "e3100fd718cb8dcbdcc16cc42ed0cd03"
server_key_iv: "9e03d9441984fcbab77e22e382b9bea2"
server_key_version: "encryptionkey@0"
I20260924 03:10:21.396826 9938 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 03:10:21.401302 9952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:21.403750 9938 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:10:21.403993 9938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "89ec5d0fec174b1686042fc999955d19"
format_stamp: "Formatted at 2026-09-24 03:10:21 on dist-test-slave-5l98"
server_key: "e3100fd718cb8dcbdcc16cc42ed0cd03"
server_key_iv: "9e03d9441984fcbab77e22e382b9bea2"
server_key_version: "encryptionkey@0"
I20260924 03:10:21.404359 9938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:21.420934 9938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:21.421779 9938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:21.422061 9938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:21.423128 9938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:21.424772 9938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:21.424902 9938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:21.425027 9938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:21.425122 9938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:21.459360 9938 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33559
I20260924 03:10:21.459414 10064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33559 every 8 connection(s)
I20260924 03:10:21.460665 9938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:10:21.468288 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 9938
I20260924 03:10:21.468493 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:10:21.469051 388 external_mini_cluster.cc:1428] Setting key c93a25fd32e1a7e1f6eb46ee04fae729
I20260924 03:10:21.471764 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45657
--builtin_ntp_servers=127.0.97.20:34807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 03:10:21.474527 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:21.463304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34169 (local address 127.0.97.62:45657)
0924 03:10:21.463488 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:21.463492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:21.465141 (+ 1649us) server_negotiation.cc:407] Connection header received
0924 03:10:21.466889 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:21.466894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:21.466960 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:21.467053 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:21.468459 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:21.469517 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:21.471554 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:21.472098 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:21.472257 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:21.473685 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:21.473691 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:21.473696 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:10:21.473725 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:21.473755 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:10:21.473767 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:21.473863 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:21.474093 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 03:10:21.474186 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 03:10:21.475550 10067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:21.463917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45657 (local address 127.0.97.2:34169)
0924 03:10:21.464924 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:21.464975 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:21.466612 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:21.467326 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:21.467334 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:21.467416 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:21.468202 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:21.468230 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:21.469685 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:21.469690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:21.471399 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:21.471411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:21.472330 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:21.472337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:21.472633 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:21.473363 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:10:21.473365 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:21.473405 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:10:21.473484 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:21.473862 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:21.473874 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:21.473885 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 03:10:21.474085 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 03:10:21.474475 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":293,"threads_started":1}
I20260924 03:10:21.477048 10065 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45657
I20260924 03:10:21.477367 10065 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:21.478124 10065 heartbeater.cc:511] Master 127.0.97.62:45657 requested a full tablet report, sending...
I20260924 03:10:21.479494 9748 ts_manager.cc:194] Registered new tserver with Master: 89ec5d0fec174b1686042fc999955d19 (127.0.97.2:33559)
I20260924 03:10:21.480293 9748 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:34169
W20260924 03:10:21.799355 10068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:21.799686 10068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:21.799841 10068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:21.807037 10068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:21.807201 10068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:21.807379 10068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:10:21.816780 10068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:10:21.818167 10068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:21.819901 10068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:21.833465 10076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:21.833552 10073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:21.833549 10074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:21.834393 10068 server_base.cc:1034] running on GCE node
I20260924 03:10:21.835187 10068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:21.836261 10068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:21.837497 10068 hybrid_clock.cc:648] HybridClock initialized: now 1790219421837448 us; error 48 us; skew 500 ppm
I20260924 03:10:21.837831 10068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:10:21.840230 10068 webserver.cc:460] Webserver started at http://127.0.97.3:36851/ using document root <none> and password file <none>
I20260924 03:10:21.841073 10068 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:21.841315 10068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:21.841683 10068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:21.844564 10068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "69e89c36d6084ec7815159e759235678"
format_stamp: "Formatted at 2026-09-24 03:10:21 on dist-test-slave-5l98"
server_key: "9770b84bdd6865474817a19d371356a0"
server_key_iv: "e39d06f6ca96203ee1047612a82697d3"
server_key_version: "encryptionkey@0"
I20260924 03:10:21.845363 10068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "69e89c36d6084ec7815159e759235678"
format_stamp: "Formatted at 2026-09-24 03:10:21 on dist-test-slave-5l98"
server_key: "9770b84bdd6865474817a19d371356a0"
server_key_iv: "e39d06f6ca96203ee1047612a82697d3"
server_key_version: "encryptionkey@0"
I20260924 03:10:21.850410 10068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:10:21.854096 10082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:21.855584 10068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:10:21.855794 10068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "69e89c36d6084ec7815159e759235678"
format_stamp: "Formatted at 2026-09-24 03:10:21 on dist-test-slave-5l98"
server_key: "9770b84bdd6865474817a19d371356a0"
server_key_iv: "e39d06f6ca96203ee1047612a82697d3"
server_key_version: "encryptionkey@0"
I20260924 03:10:21.856024 10068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:21.887249 10068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:21.888144 10068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:21.888441 10068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:21.889461 10068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:21.890939 10068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:21.891075 10068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:21.891218 10068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:21.891309 10068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:21.926545 10068 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42263
I20260924 03:10:21.926641 10194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42263 every 8 connection(s)
I20260924 03:10:21.928071 10068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:10:21.936198 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 10068
I20260924 03:10:21.936401 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:10:21.937036 388 external_mini_cluster.cc:1428] Setting key bd5a9261f7424f6d623d8bb71d397c8a
I20260924 03:10:21.942855 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:21.931012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49805 (local address 127.0.97.62:45657)
0924 03:10:21.931198 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:21.931205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:21.933233 (+ 2028us) server_negotiation.cc:407] Connection header received
0924 03:10:21.935058 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:21.935064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:21.935156 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:21.935278 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:21.936734 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:21.937655 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:21.939864 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:21.940292 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:21.940435 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:21.942006 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:21.942011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:21.942015 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:10:21.942032 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:21.942064 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:10:21.942076 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:21.942201 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:21.942433 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 03:10:21.942528 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:10:21.943994 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:21.931746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45657 (local address 127.0.97.3:49805)
0924 03:10:21.933015 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:21.933080 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:21.934766 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:21.935532 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:21.935540 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:21.935633 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:21.936404 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:21.936440 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:21.937867 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:21.937871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:21.939736 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:21.939751 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:21.940552 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:21.940560 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:21.940808 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:21.941485 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:10:21.941488 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:21.941530 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:10:21.941612 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:21.942201 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:21.942214 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:21.942224 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 03:10:21.942412 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 03:10:21.942842 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":474,"threads_started":1}
I20260924 03:10:21.945359 10195 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45657
I20260924 03:10:21.945683 10195 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:21.946442 10195 heartbeater.cc:511] Master 127.0.97.62:45657 requested a full tablet report, sending...
I20260924 03:10:21.948066 9748 ts_manager.cc:194] Registered new tserver with Master: 69e89c36d6084ec7815159e759235678 (127.0.97.3:42263)
I20260924 03:10:21.948995 9748 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:49805
I20260924 03:10:21.952741 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 03:10:21.959046 9936 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:21.957403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57928 (local address 127.0.97.62:45657)
0924 03:10:21.957696 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:21.957705 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:21.958037 (+ 332us) server_negotiation.cc:407] Connection header received
0924 03:10:21.958151 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:21.958155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:21.958226 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:21.958311 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:21.958853 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57928: BlockingRecv error: recv got EOF from 127.0.0.1:57928 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 03:10:21.959409 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 9807
I20260924 03:10:21.969429 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:21.960464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44006 (local address 127.0.97.1:38615)
0924 03:10:21.961007 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:21.961014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:21.961039 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:10:21.961095 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:21.961101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:21.961330 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:21.961450 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:21.961958 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:21.963056 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:21.964883 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:21.965858 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:21.965983 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:21.967256 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:21.967265 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:21.967280 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:10:21.968071 (+ 791us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:21.968205 (+ 134us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:10:21.968229 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:21.968341 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:21.968642 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 03:10:21.968797 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":276,"threads_started":1}
I20260924 03:10:21.970906 9929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:10:22.053184 9934 heartbeater.cc:503] Master 127.0.97.62:45657 was elected leader, sending a full tablet report...
I20260924 03:10:22.054277 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 9807
W20260924 03:10:22.071288 9733 connection.cc:535] server connection from 127.0.97.1:52389 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 03:10:22.071604 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 9938
I20260924 03:10:22.079932 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:22.072449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38424 (local address 127.0.97.2:33559)
0924 03:10:22.073033 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:22.073040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:22.073069 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:10:22.073136 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:22.073142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:22.073374 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:22.073512 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:22.073915 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:22.074934 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:22.076401 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:22.077228 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:22.077333 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:22.078361 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:22.078370 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:22.078383 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:10:22.079095 (+ 712us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:22.079141 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:10:22.079161 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:22.079260 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:22.079488 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 03:10:22.079599 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":280,"threads_started":1}
I20260924 03:10:22.081441 10060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:10:22.165606 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 9938
I20260924 03:10:22.181582 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 10068
I20260924 03:10:22.190461 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:22.182776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45460 (local address 127.0.97.3:42263)
0924 03:10:22.183430 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:22.183437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:22.183468 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:10:22.183569 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:22.183575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:22.183793 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:22.183925 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:22.184390 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:22.185532 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:22.186839 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:22.187489 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:22.187609 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:22.188889 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:22.188897 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:22.188911 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:10:22.189534 (+ 623us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:22.189589 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:10:22.189608 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:22.189702 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:22.189881 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 03:10:22.190062 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":364,"threads_started":1}
I20260924 03:10:22.191886 10190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:10:22.283154 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 10068
I20260924 03:10:22.299374 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 9717
I20260924 03:10:22.300732 9778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:10:22.405310 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 9717
2026-09-24T03:10:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2531 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T03:10:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:10:22Z Disabled control of system clock
I20260924 03:10:22.477460 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37583
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 03:10:22.823077 10214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:22.823463 10214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:22.823668 10214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:22.831782 10214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:10:22.831902 10214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:22.831987 10214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:22.832077 10214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:10:22.832139 10214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:10:22.841640 10214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:10:22.843016 10214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:22.844882 10214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:22.857393 10222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:22.857503 10220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:22.857545 10219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:22.858361 10214 server_base.cc:1034] running on GCE node
I20260924 03:10:22.859269 10214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:22.860796 10214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:22.862067 10214 hybrid_clock.cc:648] HybridClock initialized: now 1790219422862022 us; error 39 us; skew 500 ppm
I20260924 03:10:22.862498 10214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:10:22.865013 10214 webserver.cc:460] Webserver started at http://127.0.97.62:33723/ using document root <none> and password file <none>
I20260924 03:10:22.865728 10214 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:22.865862 10214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:22.866189 10214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:22.869263 10214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "a1645866d4aa49f296130641fdcaa968"
format_stamp: "Formatted at 2026-09-24 03:10:22 on dist-test-slave-5l98"
server_key: "d3fcec21a9a9ef05d5b2d9c39fc08ec8"
server_key_iv: "9cc74186aff99530656b1973126e9dd4"
server_key_version: "encryptionkey@0"
I20260924 03:10:22.870090 10214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "a1645866d4aa49f296130641fdcaa968"
format_stamp: "Formatted at 2026-09-24 03:10:22 on dist-test-slave-5l98"
server_key: "d3fcec21a9a9ef05d5b2d9c39fc08ec8"
server_key_iv: "9cc74186aff99530656b1973126e9dd4"
server_key_version: "encryptionkey@0"
I20260924 03:10:22.875640 10214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:10:22.879464 10228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:22.881211 10214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:10:22.881435 10214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "a1645866d4aa49f296130641fdcaa968"
format_stamp: "Formatted at 2026-09-24 03:10:22 on dist-test-slave-5l98"
server_key: "d3fcec21a9a9ef05d5b2d9c39fc08ec8"
server_key_iv: "9cc74186aff99530656b1973126e9dd4"
server_key_version: "encryptionkey@0"
I20260924 03:10:22.881680 10214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:22.888233 10214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:22.889110 10214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:22.889473 10214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:22.909653 10214 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37583
I20260924 03:10:22.909688 10279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37583 every 8 connection(s)
I20260924 03:10:22.911137 10214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:10:22.914920 10280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:22.920418 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 10214
I20260924 03:10:22.920723 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:10:22.921367 388 external_mini_cluster.cc:1428] Setting key f9d6c60b8383c52fff98f3e9b5eaa4e2
I20260924 03:10:22.925998 10280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968: Bootstrap starting.
I20260924 03:10:22.930105 10280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:22.930929 10283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:22.923488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60660 (local address 127.0.97.62:37583)
0924 03:10:22.924483 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:22.924527 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:22.924581 (+ 54us) server_negotiation.cc:407] Connection header received
0924 03:10:22.926812 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:22.926838 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:22.927330 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:22.927798 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:22.928846 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:22.928853 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:22.928869 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:10:22.928886 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:22.928973 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:10:22.928995 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:22.929214 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:22.929417 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 03:10:22.929673 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":297,"threads_started":1}
I20260924 03:10:22.931463 10280 log.cc:824] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:22.934705 10280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968: No bootstrap required, opened a new log
I20260924 03:10:22.942042 10280 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1645866d4aa49f296130641fdcaa968" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37583 } }
I20260924 03:10:22.942413 10280 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:22.942508 10280 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1645866d4aa49f296130641fdcaa968, State: Initialized, Role: FOLLOWER
I20260924 03:10:22.943058 10280 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1645866d4aa49f296130641fdcaa968" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37583 } }
I20260924 03:10:22.943289 10280 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:10:22.943418 10280 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:10:22.943581 10280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:22.945797 10280 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1645866d4aa49f296130641fdcaa968" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37583 } }
I20260924 03:10:22.946363 10280 leader_election.cc:302] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1645866d4aa49f296130641fdcaa968; no voters:
I20260924 03:10:22.947062 10280 leader_election.cc:288] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:10:22.947182 10285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:10:22.947960 10285 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968 [term 1 LEADER]: Becoming Leader. State: Replica: a1645866d4aa49f296130641fdcaa968, State: Running, Role: LEADER
I20260924 03:10:22.948625 10285 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1645866d4aa49f296130641fdcaa968" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37583 } }
I20260924 03:10:22.949509 10280 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:10:22.951260 10287 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1645866d4aa49f296130641fdcaa968" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1645866d4aa49f296130641fdcaa968" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37583 } } }
I20260924 03:10:22.952240 10287 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968 [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:22.951893 10286 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1645866d4aa49f296130641fdcaa968. Latest consensus state: current_term: 1 leader_uuid: "a1645866d4aa49f296130641fdcaa968" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1645866d4aa49f296130641fdcaa968" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37583 } } }
I20260924 03:10:22.952841 10286 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968 [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:22.953852 10292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:10:22.959172 10292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:10:22.968051 10292 catalog_manager.cc:1347] Generated new cluster ID: 2d68f58eb268454188cb65af5bdddfc1
I20260924 03:10:22.968175 10292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:10:22.987730 10292 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:10:22.988296 10292 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:10:23.004130 10292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1645866d4aa49f296130641fdcaa968: Generated new TSK 0
I20260924 03:10:23.005103 10292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:10:23.016835 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37583
--builtin_ntp_servers=127.0.97.20:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 03:10:23.376307 10304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:23.376741 10304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:23.376888 10304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:23.384527 10304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:23.384689 10304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:23.384871 10304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:10:23.394644 10304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:10:23.395946 10304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:23.397692 10304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:23.410859 10309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:23.410952 10310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:23.411041 10312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:23.411947 10304 server_base.cc:1034] running on GCE node
I20260924 03:10:23.412801 10304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:23.413777 10304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:23.415025 10304 hybrid_clock.cc:648] HybridClock initialized: now 1790219423415002 us; error 45 us; skew 500 ppm
I20260924 03:10:23.415376 10304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:10:23.418015 10304 webserver.cc:460] Webserver started at http://127.0.97.1:44749/ using document root <none> and password file <none>
I20260924 03:10:23.418761 10304 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:23.418900 10304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:23.419262 10304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:23.422245 10304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "a5dc7dc04b2a4962ba1a3f598546876e"
format_stamp: "Formatted at 2026-09-24 03:10:23 on dist-test-slave-5l98"
server_key: "e237ec89e4789d46fd44dd4c64669324"
server_key_iv: "97ef80a0a408e7094f4699231d7657b3"
server_key_version: "encryptionkey@0"
I20260924 03:10:23.423049 10304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a5dc7dc04b2a4962ba1a3f598546876e"
format_stamp: "Formatted at 2026-09-24 03:10:23 on dist-test-slave-5l98"
server_key: "e237ec89e4789d46fd44dd4c64669324"
server_key_iv: "97ef80a0a408e7094f4699231d7657b3"
server_key_version: "encryptionkey@0"
I20260924 03:10:23.428625 10304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 03:10:23.432773 10318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:23.434449 10304 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:10:23.434665 10304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "a5dc7dc04b2a4962ba1a3f598546876e"
format_stamp: "Formatted at 2026-09-24 03:10:23 on dist-test-slave-5l98"
server_key: "e237ec89e4789d46fd44dd4c64669324"
server_key_iv: "97ef80a0a408e7094f4699231d7657b3"
server_key_version: "encryptionkey@0"
I20260924 03:10:23.434922 10304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:23.441418 10304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:23.442332 10304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:23.442641 10304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:23.443641 10304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:23.445183 10304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:23.445307 10304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:23.445446 10304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:23.445535 10304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:23.480340 10304 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40679
I20260924 03:10:23.480397 10430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40679 every 8 connection(s)
I20260924 03:10:23.481807 10304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:10:23.483958 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 10304
I20260924 03:10:23.484294 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:10:23.485108 388 external_mini_cluster.cc:1428] Setting key c81dc6a3ce52b76cd76ef7664e4cb90e
I20260924 03:10:23.487896 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37583
--builtin_ntp_servers=127.0.97.20:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 03:10:23.491181 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:23.484525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60369 (local address 127.0.97.62:37583)
0924 03:10:23.485065 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:23.485070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:23.486466 (+ 1396us) server_negotiation.cc:407] Connection header received
0924 03:10:23.488471 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:23.488483 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:23.488575 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:23.488721 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:23.490275 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:23.490282 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:23.490287 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:10:23.490307 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:23.490363 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:10:23.490376 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:23.490512 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:23.490749 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 03:10:23.490822 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":273,"threads_started":1}
I20260924 03:10:23.492139 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:23.485139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37583 (local address 127.0.97.1:60369)
0924 03:10:23.486323 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:23.486361 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:23.488206 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:23.489006 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:23.489014 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:23.489139 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:23.489952 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:10:23.489955 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:23.489996 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:10:23.490088 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:23.490483 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:23.490494 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:23.490502 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 03:10:23.490833 (+ 331us) client_negotiation.cc:240] Negotiation successful
0924 03:10:23.491103 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":416,"threads_started":1}
I20260924 03:10:23.493867 10431 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37583
I20260924 03:10:23.494215 10431 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:23.494956 10431 heartbeater.cc:511] Master 127.0.97.62:37583 requested a full tablet report, sending...
I20260924 03:10:23.497309 10245 ts_manager.cc:194] Registered new tserver with Master: a5dc7dc04b2a4962ba1a3f598546876e (127.0.97.1:40679)
W20260924 03:10:23.823025 10435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:23.823347 10435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:23.823503 10435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:23.831235 10435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:23.831344 10435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:23.831521 10435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:10:23.840880 10435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:10:23.842249 10435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:23.843993 10435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:23.857694 10443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:23.857980 10440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:23.857726 10441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:23.858703 10435 server_base.cc:1034] running on GCE node
I20260924 03:10:23.859314 10435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:23.860311 10435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:23.861524 10435 hybrid_clock.cc:648] HybridClock initialized: now 1790219423861483 us; error 69 us; skew 500 ppm
I20260924 03:10:23.861922 10435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:10:23.864396 10435 webserver.cc:460] Webserver started at http://127.0.97.2:45427/ using document root <none> and password file <none>
I20260924 03:10:23.865170 10435 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:23.865288 10435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:23.865633 10435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:23.868623 10435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "ee3b7349c19d456fa572038a47ff8267"
format_stamp: "Formatted at 2026-09-24 03:10:23 on dist-test-slave-5l98"
server_key: "569cf176023ae146686abbde8baa5e8a"
server_key_iv: "42248193a86806e9cdec5d19933b14bb"
server_key_version: "encryptionkey@0"
I20260924 03:10:23.869453 10435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ee3b7349c19d456fa572038a47ff8267"
format_stamp: "Formatted at 2026-09-24 03:10:23 on dist-test-slave-5l98"
server_key: "569cf176023ae146686abbde8baa5e8a"
server_key_iv: "42248193a86806e9cdec5d19933b14bb"
server_key_version: "encryptionkey@0"
I20260924 03:10:23.874711 10435 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:10:23.878356 10449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:23.879921 10435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:10:23.880101 10435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "ee3b7349c19d456fa572038a47ff8267"
format_stamp: "Formatted at 2026-09-24 03:10:23 on dist-test-slave-5l98"
server_key: "569cf176023ae146686abbde8baa5e8a"
server_key_iv: "42248193a86806e9cdec5d19933b14bb"
server_key_version: "encryptionkey@0"
I20260924 03:10:23.880328 10435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:23.887221 10435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:23.888111 10435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:23.888417 10435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:23.889668 10435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:23.891146 10435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:23.891237 10435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:23.891325 10435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:23.891382 10435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:23.928973 10435 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39025
I20260924 03:10:23.929014 10561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39025 every 8 connection(s)
I20260924 03:10:23.930392 10435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:10:23.932616 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 10435
I20260924 03:10:23.932811 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:10:23.933413 388 external_mini_cluster.cc:1428] Setting key 7cb6db5c2810cb6c424091f4a18074a0
I20260924 03:10:23.936843 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37583
--builtin_ntp_servers=127.0.97.20:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 03:10:23.945468 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:23.933317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42149 (local address 127.0.97.62:37583)
0924 03:10:23.933547 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:23.933555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:23.935269 (+ 1714us) server_negotiation.cc:407] Connection header received
0924 03:10:23.938181 (+ 2912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:23.938189 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:23.938255 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:23.938358 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:23.944586 (+ 6228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:23.944595 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:23.944600 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:10:23.944629 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:23.944662 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:10:23.944676 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:23.944826 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:23.945181 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 03:10:23.945238 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 03:10:23.947034 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:23.933860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37583 (local address 127.0.97.2:42149)
0924 03:10:23.935104 (+ 1244us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:23.935149 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:23.937823 (+ 2674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:23.938709 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:23.938724 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:23.938899 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:23.944146 (+ 5247us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:10:23.944149 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:23.944202 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:10:23.944321 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:23.944827 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:23.944843 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:23.944855 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 03:10:23.945268 (+ 413us) client_negotiation.cc:240] Negotiation successful
0924 03:10:23.945606 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":501,"threads_started":1}
I20260924 03:10:23.948694 10562 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37583
I20260924 03:10:23.949060 10562 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:23.949729 10562 heartbeater.cc:511] Master 127.0.97.62:37583 requested a full tablet report, sending...
I20260924 03:10:23.951223 10245 ts_manager.cc:194] Registered new tserver with Master: ee3b7349c19d456fa572038a47ff8267 (127.0.97.2:39025)
W20260924 03:10:24.291286 10565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:24.291653 10565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:24.291827 10565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:24.299158 10565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:24.299337 10565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:24.299521 10565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:10:24.309531 10565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:10:24.310922 10565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:24.312821 10565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:24.327035 10571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:24.327170 10570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:24.327114 10573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:24.327924 10565 server_base.cc:1034] running on GCE node
I20260924 03:10:24.328800 10565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:24.329913 10565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:24.331169 10565 hybrid_clock.cc:648] HybridClock initialized: now 1790219424331096 us; error 58 us; skew 500 ppm
I20260924 03:10:24.331570 10565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:10:24.334201 10565 webserver.cc:460] Webserver started at http://127.0.97.3:43753/ using document root <none> and password file <none>
I20260924 03:10:24.334882 10565 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:24.334973 10565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:24.335263 10565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:24.338568 10565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "f236e29b6069435786e3ae617887db86"
format_stamp: "Formatted at 2026-09-24 03:10:24 on dist-test-slave-5l98"
server_key: "1e4f0404f86e9e072461e09eb9578ad0"
server_key_iv: "d7b896287cd1c4e71e098baf05e67f5d"
server_key_version: "encryptionkey@0"
I20260924 03:10:24.339428 10565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f236e29b6069435786e3ae617887db86"
format_stamp: "Formatted at 2026-09-24 03:10:24 on dist-test-slave-5l98"
server_key: "1e4f0404f86e9e072461e09eb9578ad0"
server_key_iv: "d7b896287cd1c4e71e098baf05e67f5d"
server_key_version: "encryptionkey@0"
I20260924 03:10:24.345510 10565 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 03:10:24.349710 10579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:24.351398 10565 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:10:24.351548 10565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "f236e29b6069435786e3ae617887db86"
format_stamp: "Formatted at 2026-09-24 03:10:24 on dist-test-slave-5l98"
server_key: "1e4f0404f86e9e072461e09eb9578ad0"
server_key_iv: "d7b896287cd1c4e71e098baf05e67f5d"
server_key_version: "encryptionkey@0"
I20260924 03:10:24.351778 10565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:24.357986 10565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:24.358770 10565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:24.359010 10565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:24.359946 10565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:24.361457 10565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:24.361544 10565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:24.361617 10565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:24.361677 10565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:24.398058 10565 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42269
I20260924 03:10:24.398101 10691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42269 every 8 connection(s)
I20260924 03:10:24.399379 10565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:10:24.403390 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 10565
I20260924 03:10:24.403577 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:10:24.404081 388 external_mini_cluster.cc:1428] Setting key 34652e2ed244b42d0e4bcab4937da0fa
I20260924 03:10:24.408169 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:24.402015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58337 (local address 127.0.97.62:37583)
0924 03:10:24.402234 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:24.402241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:24.403897 (+ 1656us) server_negotiation.cc:407] Connection header received
0924 03:10:24.405760 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:24.405765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:24.405832 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:24.405959 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:24.407424 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:24.407430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:24.407435 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:10:24.407454 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:24.407495 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:10:24.407507 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:24.407632 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:24.407828 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 03:10:24.407897 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 03:10:24.409227 10694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:24.402655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37583 (local address 127.0.97.3:58337)
0924 03:10:24.403680 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:24.403739 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:24.405472 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:24.406183 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:24.406191 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:24.406287 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:24.407122 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:10:24.407125 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:24.407164 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:10:24.407261 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:24.407601 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:24.407613 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:24.407621 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 03:10:24.407913 (+ 292us) client_negotiation.cc:240] Negotiation successful
0924 03:10:24.408170 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":315,"threads_started":1}
I20260924 03:10:24.410568 10692 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37583
I20260924 03:10:24.410889 10692 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:24.411451 10692 heartbeater.cc:511] Master 127.0.97.62:37583 requested a full tablet report, sending...
I20260924 03:10:24.412907 10245 ts_manager.cc:194] Registered new tserver with Master: f236e29b6069435786e3ae617887db86 (127.0.97.3:42269)
I20260924 03:10:24.419688 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 03:10:24.426437 10433 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:24.424676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60674 (local address 127.0.97.62:37583)
0924 03:10:24.424905 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:24.424915 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:24.425433 (+ 518us) server_negotiation.cc:407] Connection header received
0924 03:10:24.425600 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:24.425606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:24.425677 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:24.425767 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:24.426188 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60674: BlockingRecv error: recv got EOF from 127.0.0.1:60674 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 03:10:24.427256 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 10304
I20260924 03:10:24.431072 10701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:24.428221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53722 (local address 127.0.97.1:40679)
0924 03:10:24.428772 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:24.428779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:24.428804 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:10:24.428859 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:24.428865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:24.429082 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:24.429230 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:24.430336 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:24.430343 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:24.430354 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:10:24.430376 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:24.430422 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:10:24.430437 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:24.430547 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:24.430752 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 03:10:24.430834 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":243,"threads_started":1}
I20260924 03:10:24.432266 10426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:10:24.514317 10431 heartbeater.cc:503] Master 127.0.97.62:37583 was elected leader, sending a full tablet report...
I20260924 03:10:24.515394 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 10304
W20260924 03:10:24.531324 10231 connection.cc:535] server connection from 127.0.97.1:60369 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 03:10:24.531606 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 10435
I20260924 03:10:24.535331 10703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:24.532644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.0.97.2:39025)
0924 03:10:24.533129 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:24.533138 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:24.533164 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:10:24.533224 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:24.533230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:24.533446 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:24.533656 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:24.534649 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:24.534656 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:24.534667 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:10:24.534684 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:24.534722 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:10:24.534736 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:24.534878 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:24.534980 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 03:10:24.535080 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":236,"threads_started":1}
I20260924 03:10:24.536654 10557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:10:24.617674 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 10435
I20260924 03:10:24.633407 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 10565
I20260924 03:10:24.637202 10705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:24.634383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53598 (local address 127.0.97.3:42269)
0924 03:10:24.635046 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:24.635054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:24.635083 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:10:24.635187 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:24.635194 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:24.635419 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:24.635542 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:24.636419 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:24.636426 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:24.636437 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:10:24.636451 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:24.636551 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:10:24.636577 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:24.636710 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:24.636761 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 03:10:24.636871 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":360,"threads_started":1}
I20260924 03:10:24.638463 10687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:10:24.722128 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 10565
I20260924 03:10:24.737799 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 10214
I20260924 03:10:24.739144 10275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:10:24.844362 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 10214
2026-09-24T03:10:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2438 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:10:24 dist-test-slave-5l98 krb5kdc[10710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:10:24 dist-test-slave-5l98 krb5kdc[10710](info): set up 2 sockets
Sep 24 03:10:24 dist-test-slave-5l98 krb5kdc[10710](info): commencing operation
krb5kdc: starting...
W20260924 03:10:25.969554 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:10:26 dist-test-slave-5l98 krb5kdc[10710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:10:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:10:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:10:26.217865 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45773
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:26.551961 10723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:26.552246 10723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:26.552408 10723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:26.559636 10723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:10:26.559746 10723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:26.559832 10723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:26.559937 10723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:10:26.560020 10723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:10:26.569802 10723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:10:26.571149 10723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:26.572412 10723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:26.584285 10728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:26.584299 10731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:26.584306 10729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:26.585203 10723 server_base.cc:1034] running on GCE node
I20260924 03:10:26.585991 10723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:26.587435 10723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:26.588660 10723 hybrid_clock.cc:648] HybridClock initialized: now 1790219426588628 us; error 64 us; skew 500 ppm
I20260924 03:10:26.589035 10723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:10:26 dist-test-slave-5l98 krb5kdc[10710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:26.592221 10723 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:10:26.593976 10723 webserver.cc:460] Webserver started at http://127.0.97.62:38191/ using document root <none> and password file <none>
I20260924 03:10:26.594660 10723 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:26.594805 10723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:26.595095 10723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:26.598040 10723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "8035f8517c89495a94900b92183fa41e"
format_stamp: "Formatted at 2026-09-24 03:10:26 on dist-test-slave-5l98"
server_key: "5d064f4fc2006b896100c07f9e871282"
server_key_iv: "8899f2719975a4a1f33213cb19a2dfd1"
server_key_version: "encryptionkey@0"
I20260924 03:10:26.598809 10723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "8035f8517c89495a94900b92183fa41e"
format_stamp: "Formatted at 2026-09-24 03:10:26 on dist-test-slave-5l98"
server_key: "5d064f4fc2006b896100c07f9e871282"
server_key_iv: "8899f2719975a4a1f33213cb19a2dfd1"
server_key_version: "encryptionkey@0"
I20260924 03:10:26.604079 10723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 03:10:26.607595 10738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:26.609050 10723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:10:26.609251 10723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "8035f8517c89495a94900b92183fa41e"
format_stamp: "Formatted at 2026-09-24 03:10:26 on dist-test-slave-5l98"
server_key: "5d064f4fc2006b896100c07f9e871282"
server_key_iv: "8899f2719975a4a1f33213cb19a2dfd1"
server_key_version: "encryptionkey@0"
I20260924 03:10:26.609452 10723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:26.626520 10723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:26.630993 10723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:26.631297 10723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:26.650388 10723 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45773
I20260924 03:10:26.650437 10789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45773 every 8 connection(s)
I20260924 03:10:26.651774 10723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:10:26.655457 10790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:26.661280 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 10723
I20260924 03:10:26.661515 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:10:26.662153 388 external_mini_cluster.cc:1428] Setting key 772c6565e82a41a34b2aea55b4ad38a8
I20260924 03:10:26.666893 10790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e: Bootstrap starting.
I20260924 03:10:26.670684 10790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:26.671979 10790 log.cc:824] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:26.675628 10790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e: No bootstrap required, opened a new log
Sep 24 03:10:26 dist-test-slave-5l98 krb5kdc[10710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:10:26.685896 10790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8035f8517c89495a94900b92183fa41e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45773 } }
I20260924 03:10:26.686396 10790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:26.686514 10790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8035f8517c89495a94900b92183fa41e, State: Initialized, Role: FOLLOWER
I20260924 03:10:26.687191 10790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8035f8517c89495a94900b92183fa41e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45773 } }
I20260924 03:10:26.687451 10790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:10:26.687587 10790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:10:26.687765 10790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:26.687860 10793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:26.664313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44050 (local address 127.0.97.62:45773)
0924 03:10:26.665108 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:26.665123 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:26.665168 (+ 45us) server_negotiation.cc:407] Connection header received
0924 03:10:26.667477 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:26.667503 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:26.667964 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:26.668382 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:26.669729 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:26.671735 (+ 2006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:26.673510 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:26.674679 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:26.675048 (+ 369us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:26.679340 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:26.679352 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:26.679374 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:26.679424 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:26.683369 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:26.684079 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:26.684090 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:26.684103 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:26.684309 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:26.684710 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:26.684716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:26.684719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:26.685451 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:26.685750 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:26.686066 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 03:10:26.686499 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":354,"threads_started":1}
I20260924 03:10:26.690584 10790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8035f8517c89495a94900b92183fa41e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45773 } }
I20260924 03:10:26.691077 10790 leader_election.cc:302] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8035f8517c89495a94900b92183fa41e; no voters:
I20260924 03:10:26.691635 10790 leader_election.cc:288] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:10:26.691764 10795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:10:26.693336 10795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e [term 1 LEADER]: Becoming Leader. State: Replica: 8035f8517c89495a94900b92183fa41e, State: Running, Role: LEADER
I20260924 03:10:26.693790 10790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:10:26.693845 10795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8035f8517c89495a94900b92183fa41e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45773 } }
I20260924 03:10:26.696516 10797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8035f8517c89495a94900b92183fa41e. Latest consensus state: current_term: 1 leader_uuid: "8035f8517c89495a94900b92183fa41e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8035f8517c89495a94900b92183fa41e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45773 } } }
I20260924 03:10:26.696844 10797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:26.696723 10796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8035f8517c89495a94900b92183fa41e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8035f8517c89495a94900b92183fa41e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45773 } } }
I20260924 03:10:26.697113 10796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:26.699357 10801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:10:26.704540 10801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:10:26.714491 10801 catalog_manager.cc:1347] Generated new cluster ID: 6bfcec19ae8c4253a9a3fafe39fba8ed
I20260924 03:10:26.714607 10801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:10:26.738754 10801 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:10:26.740137 10801 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:10:26.747330 10801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e: Generated new TSK 0
I20260924 03:10:26.748298 10801 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 03:10:26.824735 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45773
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:27.162549 10816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:27.162823 10816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:27.162984 10816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:27.171106 10816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:27.171229 10816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:27.171422 10816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:10:27.181586 10816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:10:27.182889 10816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:27.184180 10816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:27.198212 10821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:27.198560 10822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:27.198772 10824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:27.199076 10816 server_base.cc:1034] running on GCE node
I20260924 03:10:27.199774 10816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:27.200919 10816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:27.202193 10816 hybrid_clock.cc:648] HybridClock initialized: now 1790219427202088 us; error 69 us; skew 500 ppm
I20260924 03:10:27.202553 10816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:10:27 dist-test-slave-5l98 krb5kdc[10710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:27.205930 10816 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:10:27.207358 10816 webserver.cc:460] Webserver started at http://127.0.97.1:42177/ using document root <none> and password file <none>
I20260924 03:10:27.208124 10816 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:27.208333 10816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:27.208698 10816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:27.212049 10816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "4989246e85904134b7ed1a99da335c7a"
format_stamp: "Formatted at 2026-09-24 03:10:27 on dist-test-slave-5l98"
server_key: "38faff4702a07063dc9d0cf7bc2c809f"
server_key_iv: "f3ab4c6163545f18d2e78a6de4bd6d23"
server_key_version: "encryptionkey@0"
I20260924 03:10:27.212961 10816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4989246e85904134b7ed1a99da335c7a"
format_stamp: "Formatted at 2026-09-24 03:10:27 on dist-test-slave-5l98"
server_key: "38faff4702a07063dc9d0cf7bc2c809f"
server_key_iv: "f3ab4c6163545f18d2e78a6de4bd6d23"
server_key_version: "encryptionkey@0"
I20260924 03:10:27.218544 10816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:10:27.222466 10831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:27.223847 10816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:10:27.224013 10816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "4989246e85904134b7ed1a99da335c7a"
format_stamp: "Formatted at 2026-09-24 03:10:27 on dist-test-slave-5l98"
server_key: "38faff4702a07063dc9d0cf7bc2c809f"
server_key_iv: "f3ab4c6163545f18d2e78a6de4bd6d23"
server_key_version: "encryptionkey@0"
I20260924 03:10:27.224211 10816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:27.241794 10816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:27.245769 10816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:27.246053 10816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:27.247020 10816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:27.248381 10816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:27.248485 10816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:27.248677 10816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:27.248754 10816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:27.282800 10816 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44915
I20260924 03:10:27.282845 10943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44915 every 8 connection(s)
I20260924 03:10:27.284047 10816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:10:27.291560 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 10816
I20260924 03:10:27.291790 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:10:27.292455 388 external_mini_cluster.cc:1428] Setting key 12d0d56d288a5a49f6b726dd9606aab5
Sep 24 03:10:27 dist-test-slave-5l98 krb5kdc[10710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:10:27.304806 10946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:27.286608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38745 (local address 127.0.97.62:45773)
0924 03:10:27.287050 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:27.287054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:27.287837 (+ 783us) server_negotiation.cc:407] Connection header received
0924 03:10:27.289753 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:27.289759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:27.289849 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:27.289970 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:27.291490 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:27.292311 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:27.294148 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:27.294562 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:27.294727 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:27.298483 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:27.298488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:27.298491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:27.298523 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:27.301325 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:27.302128 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:27.302136 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:27.302139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:27.302253 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:27.302769 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:27.302772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:27.302775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:27.303035 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:27.303237 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:27.304251 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0924 03:10:27.304424 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":243,"threads_started":1}
I20260924 03:10:27.306249 10947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:27.286986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45773 (local address 127.0.97.1:38745)
0924 03:10:27.287689 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:27.287725 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:27.289489 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:27.290231 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:27.290239 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:27.290748 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:27.291311 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:27.291335 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:27.292473 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:27.292477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:27.293998 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:27.294010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:27.294837 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:27.294844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:27.295069 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:27.295760 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:27.295787 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:27.298248 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:27.301471 (+ 3223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:27.301481 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:27.301498 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:27.301997 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:27.302383 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:27.302392 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:27.302398 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:27.302635 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:27.303262 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:27.303274 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:27.303733 (+ 459us) client_negotiation.cc:769] Sending connection context
0924 03:10:27.304043 (+ 310us) client_negotiation.cc:240] Negotiation successful
0924 03:10:27.304654 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":293,"threads_started":1}
I20260924 03:10:27.308037 10944 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45773
I20260924 03:10:27.308559 10944 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:27.309309 10944 heartbeater.cc:511] Master 127.0.97.62:45773 requested a full tablet report, sending...
I20260924 03:10:27.311441 10755 ts_manager.cc:194] Registered new tserver with Master: 4989246e85904134b7ed1a99da335c7a (127.0.97.1:44915)
I20260924 03:10:27.313016 10755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38745
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 03:10:27.355219 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45773
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:27.683949 10950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:27.684244 10950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:27.684343 10950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:27.691763 10950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:27.691869 10950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:27.691978 10950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:10:27.701936 10950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:10:27.703132 10950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:27.704332 10950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:27.717573 10958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:27.717599 10956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:27.717686 10955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:27.718791 10950 server_base.cc:1034] running on GCE node
I20260924 03:10:27.719408 10950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:27.720309 10950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:27.721537 10950 hybrid_clock.cc:648] HybridClock initialized: now 1790219427721460 us; error 66 us; skew 500 ppm
I20260924 03:10:27.721884 10950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:10:27 dist-test-slave-5l98 krb5kdc[10710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:27.725337 10950 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:10:27.726858 10950 webserver.cc:460] Webserver started at http://127.0.97.2:42361/ using document root <none> and password file <none>
I20260924 03:10:27.727576 10950 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:27.727699 10950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:27.727993 10950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:27.731376 10950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "76858f8df2ab47938cbb4a4682abd879"
format_stamp: "Formatted at 2026-09-24 03:10:27 on dist-test-slave-5l98"
server_key: "291c753c3dfa961097fc1433650dfb93"
server_key_iv: "df7cbe250a6b34708ac335f4d778e22c"
server_key_version: "encryptionkey@0"
I20260924 03:10:27.732168 10950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "76858f8df2ab47938cbb4a4682abd879"
format_stamp: "Formatted at 2026-09-24 03:10:27 on dist-test-slave-5l98"
server_key: "291c753c3dfa961097fc1433650dfb93"
server_key_iv: "df7cbe250a6b34708ac335f4d778e22c"
server_key_version: "encryptionkey@0"
I20260924 03:10:27.737465 10950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 03:10:27.741940 10965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:27.743347 10950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260924 03:10:27.743533 10950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "76858f8df2ab47938cbb4a4682abd879"
format_stamp: "Formatted at 2026-09-24 03:10:27 on dist-test-slave-5l98"
server_key: "291c753c3dfa961097fc1433650dfb93"
server_key_iv: "df7cbe250a6b34708ac335f4d778e22c"
server_key_version: "encryptionkey@0"
I20260924 03:10:27.743769 10950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:27.756068 10950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:27.760996 10950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:27.761315 10950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:27.762329 10950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:27.763898 10950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:27.763968 10950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:27.764112 10950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:27.764163 10950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:27.799893 10950 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33433
I20260924 03:10:27.799989 11077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33433 every 8 connection(s)
I20260924 03:10:27.801270 10950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:10:27.808924 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 10950
I20260924 03:10:27.809170 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:10:27.809584 388 external_mini_cluster.cc:1428] Setting key 03365f1617d0bc3abdd63e194f27d1b9
Sep 24 03:10:27 dist-test-slave-5l98 krb5kdc[10710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:10:27.823644 10946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:27.803864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50277 (local address 127.0.97.62:45773)
0924 03:10:27.804045 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:27.804050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:27.805326 (+ 1276us) server_negotiation.cc:407] Connection header received
0924 03:10:27.807064 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:27.807071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:27.807143 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:27.807238 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:27.808873 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:27.809777 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:27.811684 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:27.812127 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:27.812267 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:27.816373 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:27.816382 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:27.816390 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:27.816438 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:27.819704 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:27.820820 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:27.820825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:27.820827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:27.820960 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:27.821579 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:27.821583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:27.821586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:27.821859 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:27.821968 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:27.823066 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0924 03:10:27.823374 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:10:27.824991 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:27.804323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45773 (local address 127.0.97.2:50277)
0924 03:10:27.805154 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:27.805202 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:27.806790 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:27.807506 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:27.807514 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:27.808007 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:27.808688 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:27.808707 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:27.809924 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:27.809935 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:27.811537 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:27.811548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:27.812353 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:27.812360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:27.812674 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:27.813406 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:27.813429 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:27.816097 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:27.819879 (+ 3782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:27.819893 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:27.819917 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:27.820620 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:27.821176 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:27.821184 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:27.821191 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:27.821443 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:27.821988 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:27.821999 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:27.822518 (+ 519us) client_negotiation.cc:769] Sending connection context
0924 03:10:27.822857 (+ 339us) client_negotiation.cc:240] Negotiation successful
0924 03:10:27.823461 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":320,"threads_started":1}
I20260924 03:10:27.826694 11078 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45773
I20260924 03:10:27.827033 11078 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:27.828135 11078 heartbeater.cc:511] Master 127.0.97.62:45773 requested a full tablet report, sending...
I20260924 03:10:27.830132 10755 ts_manager.cc:194] Registered new tserver with Master: 76858f8df2ab47938cbb4a4682abd879 (127.0.97.2:33433)
I20260924 03:10:27.831435 10755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50277
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 03:10:27.869657 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45773
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:28.245275 11083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:28.245647 11083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:28.245826 11083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:28.254334 11083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:28.254513 11083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:28.254772 11083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:10:28.264985 11083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:10:28.266368 11083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:28.267755 11083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:28.282033 11089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:28.282141 11091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:28.282155 11088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:28.289024 11083 server_base.cc:1034] running on GCE node
I20260924 03:10:28.289727 11083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:28.290885 11083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:28.292179 11083 hybrid_clock.cc:648] HybridClock initialized: now 1790219428292113 us; error 61 us; skew 500 ppm
I20260924 03:10:28.292553 11083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:10:28 dist-test-slave-5l98 krb5kdc[10710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:28.295977 11083 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:10:28.297585 11083 webserver.cc:460] Webserver started at http://127.0.97.3:37453/ using document root <none> and password file <none>
I20260924 03:10:28.298262 11083 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:28.298359 11083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:28.298593 11083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:28.301606 11083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "7d6d18babe2d41309b20280399c71d5b"
format_stamp: "Formatted at 2026-09-24 03:10:28 on dist-test-slave-5l98"
server_key: "c046dcfba65de54edc7ce536d27e6296"
server_key_iv: "462cb9b99e9cc6c323d55172d85c9721"
server_key_version: "encryptionkey@0"
I20260924 03:10:28.302305 11083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7d6d18babe2d41309b20280399c71d5b"
format_stamp: "Formatted at 2026-09-24 03:10:28 on dist-test-slave-5l98"
server_key: "c046dcfba65de54edc7ce536d27e6296"
server_key_iv: "462cb9b99e9cc6c323d55172d85c9721"
server_key_version: "encryptionkey@0"
I20260924 03:10:28.307421 11083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:10:28.311230 11098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:28.312978 11083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:10:28.313125 11083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "7d6d18babe2d41309b20280399c71d5b"
format_stamp: "Formatted at 2026-09-24 03:10:28 on dist-test-slave-5l98"
server_key: "c046dcfba65de54edc7ce536d27e6296"
server_key_iv: "462cb9b99e9cc6c323d55172d85c9721"
server_key_version: "encryptionkey@0"
I20260924 03:10:28.313400 11083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:28.316222 10944 heartbeater.cc:503] Master 127.0.97.62:45773 was elected leader, sending a full tablet report...
I20260924 03:10:28.335410 11083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:28.340370 11083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:28.340741 11083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:28.341796 11083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:28.343274 11083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:28.343355 11083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:28.343446 11083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:28.343494 11083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:28.382907 11083 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33413
I20260924 03:10:28.383014 11210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33413 every 8 connection(s)
I20260924 03:10:28.384397 11083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:10:28.388113 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 11083
I20260924 03:10:28.388366 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:10:28.389006 388 external_mini_cluster.cc:1428] Setting key ea6cf6d18c77cf64f656cf1cf85448bc
Sep 24 03:10:28 dist-test-slave-5l98 krb5kdc[10710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:10:28.407297 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:28.387225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46449 (local address 127.0.97.62:45773)
0924 03:10:28.387712 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:28.387717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:28.389119 (+ 1402us) server_negotiation.cc:407] Connection header received
0924 03:10:28.391436 (+ 2317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:28.391442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:28.391546 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:28.391673 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:28.393379 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.394223 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.395970 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.396407 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.396628 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:28.400531 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:28.400538 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:28.400543 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:28.400586 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:28.403994 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:28.404919 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:28.404923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:28.404925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:28.405115 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:28.405584 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:28.405589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:28.405591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:28.405884 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:28.406028 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:28.406777 (+ 749us) server_negotiation.cc:299] Negotiation successful
0924 03:10:28.406935 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":260,"threads_started":1}
I20260924 03:10:28.408310 11214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:28.387837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45773 (local address 127.0.97.3:46449)
0924 03:10:28.388820 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:28.388865 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:28.391118 (+ 2253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:28.391955 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:28.391968 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:28.392611 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:28.393209 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:28.393225 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.394414 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.394418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:28.395822 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:28.395834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.396718 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.396725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:28.396979 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:28.397621 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:28.397644 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:28.400286 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:28.404168 (+ 3882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:28.404178 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:28.404197 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:28.404815 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:28.405243 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:28.405247 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:28.405249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:28.405439 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:28.406028 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:28.406034 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:28.406358 (+ 324us) client_negotiation.cc:769] Sending connection context
0924 03:10:28.406635 (+ 277us) client_negotiation.cc:240] Negotiation successful
0924 03:10:28.407034 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":426,"threads_started":1}
I20260924 03:10:28.410063 11211 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45773
I20260924 03:10:28.410382 11211 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:28.411072 11211 heartbeater.cc:511] Master 127.0.97.62:45773 requested a full tablet report, sending...
I20260924 03:10:28.412691 10755 ts_manager.cc:194] Registered new tserver with Master: 7d6d18babe2d41309b20280399c71d5b (127.0.97.3:33413)
I20260924 03:10:28.413465 10755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46449
I20260924 03:10:28.417693 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:10:28.436416 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:28.422318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44058 (local address 127.0.97.62:45773)
0924 03:10:28.422576 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:28.422586 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:28.422977 (+ 391us) server_negotiation.cc:407] Connection header received
0924 03:10:28.423079 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:28.423083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:28.423150 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:28.423254 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:28.424704 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.425517 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.426926 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.427521 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.427644 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:28.430490 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:28.430496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:28.430501 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:28.430548 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:28.433265 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:28.434146 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:28.434150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:28.434153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:28.434294 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:28.434809 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:28.434814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:28.434816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:28.435269 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:28.435479 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:28.435822 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 03:10:28.436002 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 03:10:28.441167 10755 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:10:28.448585 10755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:10:28.472961 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:28.462229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52048 (local address 127.0.97.2:33433)
0924 03:10:28.462882 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:28.462894 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:28.463581 (+ 687us) server_negotiation.cc:407] Connection header received
0924 03:10:28.464286 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:28.464298 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:28.464622 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:28.464822 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:28.465933 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.467570 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.470387 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.471674 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.471840 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:28.472060 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:28.472386 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 03:10:28.472602 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":347,"threads_started":1}
I20260924 03:10:28.473073 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:28.462059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33433 (local address 127.0.0.1:52048)
0924 03:10:28.463389 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:28.463467 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:28.464061 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:28.465155 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:28.465165 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:28.465202 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:28.465730 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:28.465744 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.467712 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.467721 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:28.470197 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:28.470216 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.471891 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.471898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:28.472134 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:28.472173 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 03:10:28.472329 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 03:10:28.472564 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1017,"mutex_wait_us":36,"spinlock_wait_cycles":10624,"thread_start_us":981,"threads_started":1}
I20260924 03:10:28.474192 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:28.463036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59636 (local address 127.0.97.3:33413)
0924 03:10:28.464480 (+ 1444us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:28.464519 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:28.464556 (+ 37us) server_negotiation.cc:407] Connection header received
0924 03:10:28.464652 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:28.464662 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:28.464903 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:28.465093 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:28.466455 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.468153 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.471040 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.472604 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.472740 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:28.472979 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:28.473789 (+ 810us) server_negotiation.cc:299] Negotiation successful
0924 03:10:28.473919 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1270,"thread_start_us":1091,"threads_started":1}
I20260924 03:10:28.474308 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:28.462322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33413 (local address 127.0.0.1:59636)
0924 03:10:28.463389 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:28.463647 (+ 258us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:28.463811 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:28.465404 (+ 1593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:28.465409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:28.465430 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:28.466228 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:28.466240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.468365 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.468371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:28.470850 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:28.470864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.473405 (+ 2541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.473415 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:28.473594 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:28.473630 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 03:10:28.473734 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 03:10:28.473865 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":73,"thread_start_us":1063,"threads_started":1}
I20260924 03:10:28.477258 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:28.464068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44915 (local address 127.0.0.1:36174)
0924 03:10:28.464762 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:28.464784 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:28.464904 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:28.466821 (+ 1917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:28.466827 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:28.466857 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:28.467236 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:28.467248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.469802 (+ 2554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.469808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:28.473256 (+ 3448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:28.474291 (+ 1035us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.475975 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.475986 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:28.476606 (+ 620us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:28.476649 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 03:10:28.476729 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 03:10:28.476878 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":3,"thread_start_us":1479,"threads_started":1}
I20260924 03:10:28.477427 11013 tablet_service.cc:1467] Processing CreateTablet for tablet 8f21db3892b04ef8810ef4aac5acb6c4 (DEFAULT_TABLE table=test-table [id=0b59c13dd0934d09b588d636f2ecff67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:10:28.478338 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:28.464831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36174 (local address 127.0.97.1:44915)
0924 03:10:28.466234 (+ 1403us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:28.466240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:28.466262 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:10:28.466362 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:28.466371 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:28.466601 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:28.466813 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:28.467420 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.469649 (+ 2229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.474495 (+ 4846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.475731 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.477501 (+ 1770us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:28.477753 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:28.477827 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 03:10:28.478001 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1236,"thread_start_us":1075,"threads_started":1}
I20260924 03:10:28.478976 11013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f21db3892b04ef8810ef4aac5acb6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:28.478508 11146 tablet_service.cc:1467] Processing CreateTablet for tablet 8f21db3892b04ef8810ef4aac5acb6c4 (DEFAULT_TABLE table=test-table [id=0b59c13dd0934d09b588d636f2ecff67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:10:28.481374 10879 tablet_service.cc:1467] Processing CreateTablet for tablet 8f21db3892b04ef8810ef4aac5acb6c4 (DEFAULT_TABLE table=test-table [id=0b59c13dd0934d09b588d636f2ecff67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:10:28.482503 11146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f21db3892b04ef8810ef4aac5acb6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:28.482533 10879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f21db3892b04ef8810ef4aac5acb6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:28.491411 11230 tablet_bootstrap.cc:492] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879: Bootstrap starting.
I20260924 03:10:28.495177 11230 tablet_bootstrap.cc:654] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:28.496236 11230 log.cc:824] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:28.500993 11233 tablet_bootstrap.cc:492] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a: Bootstrap starting.
I20260924 03:10:28.503268 11232 tablet_bootstrap.cc:492] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b: Bootstrap starting.
I20260924 03:10:28.505048 11230 tablet_bootstrap.cc:492] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879: No bootstrap required, opened a new log
I20260924 03:10:28.505352 11230 ts_tablet_manager.cc:1397] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.001s
I20260924 03:10:28.505596 11233 tablet_bootstrap.cc:654] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:28.507318 11233 log.cc:824] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:28.507567 11232 tablet_bootstrap.cc:654] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:28.508811 11232 log.cc:824] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:28.510573 11233 tablet_bootstrap.cc:492] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a: No bootstrap required, opened a new log
I20260924 03:10:28.510855 11233 ts_tablet_manager.cc:1397] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 03:10:28.512048 11232 tablet_bootstrap.cc:492] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b: No bootstrap required, opened a new log
I20260924 03:10:28.512342 11232 ts_tablet_manager.cc:1397] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 03:10:28.515153 11230 raft_consensus.cc:348] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76858f8df2ab47938cbb4a4682abd879" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33433 } } peers { permanent_uuid: "7d6d18babe2d41309b20280399c71d5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33413 } } peers { permanent_uuid: "4989246e85904134b7ed1a99da335c7a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44915 } }
I20260924 03:10:28.515545 11230 raft_consensus.cc:374] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:28.515651 11230 raft_consensus.cc:729] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76858f8df2ab47938cbb4a4682abd879, State: Initialized, Role: FOLLOWER
I20260924 03:10:28.516309 11230 consensus_queue.cc:260] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76858f8df2ab47938cbb4a4682abd879" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33433 } } peers { permanent_uuid: "7d6d18babe2d41309b20280399c71d5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33413 } } peers { permanent_uuid: "4989246e85904134b7ed1a99da335c7a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44915 } }
I20260924 03:10:28.523023 11233 raft_consensus.cc:348] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76858f8df2ab47938cbb4a4682abd879" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33433 } } peers { permanent_uuid: "7d6d18babe2d41309b20280399c71d5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33413 } } peers { permanent_uuid: "4989246e85904134b7ed1a99da335c7a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44915 } }
I20260924 03:10:28.523521 11233 raft_consensus.cc:374] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:28.523630 11233 raft_consensus.cc:729] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4989246e85904134b7ed1a99da335c7a, State: Initialized, Role: FOLLOWER
I20260924 03:10:28.524550 11078 heartbeater.cc:503] Master 127.0.97.62:45773 was elected leader, sending a full tablet report...
I20260924 03:10:28.524456 11232 raft_consensus.cc:348] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76858f8df2ab47938cbb4a4682abd879" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33433 } } peers { permanent_uuid: "7d6d18babe2d41309b20280399c71d5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33413 } } peers { permanent_uuid: "4989246e85904134b7ed1a99da335c7a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44915 } }
I20260924 03:10:28.524968 11232 raft_consensus.cc:374] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:28.525079 11232 raft_consensus.cc:729] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d6d18babe2d41309b20280399c71d5b, State: Initialized, Role: FOLLOWER
I20260924 03:10:28.525300 11230 ts_tablet_manager.cc:1428] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879: Time spent starting tablet: real 0.020s user 0.008s sys 0.011s
I20260924 03:10:28.525691 11233 consensus_queue.cc:260] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76858f8df2ab47938cbb4a4682abd879" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33433 } } peers { permanent_uuid: "7d6d18babe2d41309b20280399c71d5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33413 } } peers { permanent_uuid: "4989246e85904134b7ed1a99da335c7a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44915 } }
I20260924 03:10:28.526106 11232 consensus_queue.cc:260] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76858f8df2ab47938cbb4a4682abd879" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33433 } } peers { permanent_uuid: "7d6d18babe2d41309b20280399c71d5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33413 } } peers { permanent_uuid: "4989246e85904134b7ed1a99da335c7a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44915 } }
I20260924 03:10:28.527485 11233 ts_tablet_manager.cc:1428] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260924 03:10:28.528604 11211 heartbeater.cc:503] Master 127.0.97.62:45773 was elected leader, sending a full tablet report...
I20260924 03:10:28.529424 11232 ts_tablet_manager.cc:1428] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
W20260924 03:10:28.536458 10945 tablet.cc:2369] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:10:28.553292 11079 tablet.cc:2369] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:10:28.636178 11212 tablet.cc:2369] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:10:28.717314 11237 raft_consensus.cc:482] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:10:28.717706 11237 raft_consensus.cc:504] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76858f8df2ab47938cbb4a4682abd879" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33433 } } peers { permanent_uuid: "7d6d18babe2d41309b20280399c71d5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33413 } } peers { permanent_uuid: "4989246e85904134b7ed1a99da335c7a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44915 } }
I20260924 03:10:28.719810 11237 leader_election.cc:288] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76858f8df2ab47938cbb4a4682abd879 (127.0.97.2:33433), 7d6d18babe2d41309b20280399c71d5b (127.0.97.3:33413)
I20260924 03:10:28.727772 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:28.720536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45841 (local address 127.0.97.2:33433)
0924 03:10:28.720789 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:28.720795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:28.721444 (+ 649us) server_negotiation.cc:407] Connection header received
0924 03:10:28.721628 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:28.721634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:28.721709 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:28.721785 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:28.722569 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.723401 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.725765 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.726598 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.726716 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:28.726819 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:28.727352 (+ 533us) server_negotiation.cc:299] Negotiation successful
0924 03:10:28.727505 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 03:10:28.727873 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:28.720236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57909 (local address 127.0.97.3:33413)
0924 03:10:28.720481 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:28.720526 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:28.721366 (+ 840us) server_negotiation.cc:407] Connection header received
0924 03:10:28.721675 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:28.721679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:28.721736 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:28.721856 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:28.722721 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.723743 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.725856 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.726818 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.726980 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:28.727116 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:28.727469 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 03:10:28.727625 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 03:10:28.727958 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:28.720166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33413 (local address 127.0.97.1:57909)
0924 03:10:28.721203 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:28.721235 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:28.721461 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:28.721897 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:28.721901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:28.721934 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:28.722535 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:28.722550 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.723937 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.723941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:28.725656 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:28.725695 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.727116 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.727123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:28.727312 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:28.727346 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:10:28.727441 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 03:10:28.727560 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":62,"thread_start_us":582,"threads_started":1}
I20260924 03:10:28.728562 11240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:28.720215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33433 (local address 127.0.97.1:45841)
0924 03:10:28.721347 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:28.721365 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:28.721478 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:28.721829 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:28.721836 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:28.721868 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:28.722388 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:28.722398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.723567 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.723573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:28.725546 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:28.725567 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.726864 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.726884 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:28.727116 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:28.727200 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 03:10:28.727283 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 03:10:28.727462 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"mutex_wait_us":28,"thread_start_us":748,"threads_started":1}
I20260924 03:10:28.729219 11166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f21db3892b04ef8810ef4aac5acb6c4" candidate_uuid: "4989246e85904134b7ed1a99da335c7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d6d18babe2d41309b20280399c71d5b" is_pre_election: true
I20260924 03:10:28.729825 11166 raft_consensus.cc:2454] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4989246e85904134b7ed1a99da335c7a in term 0.
I20260924 03:10:28.729724 11033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f21db3892b04ef8810ef4aac5acb6c4" candidate_uuid: "4989246e85904134b7ed1a99da335c7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76858f8df2ab47938cbb4a4682abd879" is_pre_election: true
I20260924 03:10:28.730252 11033 raft_consensus.cc:2454] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4989246e85904134b7ed1a99da335c7a in term 0.
I20260924 03:10:28.730636 10832 leader_election.cc:302] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4989246e85904134b7ed1a99da335c7a, 7d6d18babe2d41309b20280399c71d5b; no voters:
I20260924 03:10:28.731106 11237 raft_consensus.cc:2781] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:10:28.731323 11237 raft_consensus.cc:482] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:10:28.731444 11237 raft_consensus.cc:3037] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:28.733774 11237 raft_consensus.cc:504] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76858f8df2ab47938cbb4a4682abd879" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33433 } } peers { permanent_uuid: "7d6d18babe2d41309b20280399c71d5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33413 } } peers { permanent_uuid: "4989246e85904134b7ed1a99da335c7a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44915 } }
I20260924 03:10:28.734449 11237 leader_election.cc:288] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a [CANDIDATE]: Term 1 election: Requested vote from peers 76858f8df2ab47938cbb4a4682abd879 (127.0.97.2:33433), 7d6d18babe2d41309b20280399c71d5b (127.0.97.3:33413)
I20260924 03:10:28.735093 11166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f21db3892b04ef8810ef4aac5acb6c4" candidate_uuid: "4989246e85904134b7ed1a99da335c7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d6d18babe2d41309b20280399c71d5b"
I20260924 03:10:28.735131 11033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f21db3892b04ef8810ef4aac5acb6c4" candidate_uuid: "4989246e85904134b7ed1a99da335c7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76858f8df2ab47938cbb4a4682abd879"
I20260924 03:10:28.735306 11166 raft_consensus.cc:3037] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:28.735396 11033 raft_consensus.cc:3037] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:28.737666 11166 raft_consensus.cc:2454] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4989246e85904134b7ed1a99da335c7a in term 1.
I20260924 03:10:28.737968 11033 raft_consensus.cc:2454] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4989246e85904134b7ed1a99da335c7a in term 1.
I20260924 03:10:28.738080 10832 leader_election.cc:302] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4989246e85904134b7ed1a99da335c7a, 7d6d18babe2d41309b20280399c71d5b; no voters:
I20260924 03:10:28.738445 11237 raft_consensus.cc:2781] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:10:28.738940 11237 raft_consensus.cc:686] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a [term 1 LEADER]: Becoming Leader. State: Replica: 4989246e85904134b7ed1a99da335c7a, State: Running, Role: LEADER
I20260924 03:10:28.739491 11237 consensus_queue.cc:237] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76858f8df2ab47938cbb4a4682abd879" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33433 } } peers { permanent_uuid: "7d6d18babe2d41309b20280399c71d5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33413 } } peers { permanent_uuid: "4989246e85904134b7ed1a99da335c7a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44915 } }
I20260924 03:10:28.744123 10753 catalog_manager.cc:5495] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4989246e85904134b7ed1a99da335c7a (127.0.97.1). New cstate: current_term: 1 leader_uuid: "4989246e85904134b7ed1a99da335c7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76858f8df2ab47938cbb4a4682abd879" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d6d18babe2d41309b20280399c71d5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4989246e85904134b7ed1a99da335c7a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44915 } health_report { overall_health: HEALTHY } } }
I20260924 03:10:28.775470 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:28.769911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36176 (local address 127.0.97.1:44915)
0924 03:10:28.770126 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:28.770134 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:28.770203 (+ 69us) server_negotiation.cc:407] Connection header received
0924 03:10:28.770466 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:28.770470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:28.770564 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:28.770773 (+ 209us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:10:28.771157 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.771889 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.773256 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.773665 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.773812 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:28.774335 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:10:28.774853 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:10:28.775017 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:28.775114 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 03:10:28.775198 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 03:10:28.784364 11166 raft_consensus.cc:1261] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b [term 1 FOLLOWER]: Refusing update from remote peer 4989246e85904134b7ed1a99da335c7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:10:28.784394 11033 raft_consensus.cc:1261] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879 [term 1 FOLLOWER]: Refusing update from remote peer 4989246e85904134b7ed1a99da335c7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:10:28.785447 11241 consensus_queue.cc:1035] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "76858f8df2ab47938cbb4a4682abd879" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:10:28.785830 11237 consensus_queue.cc:1035] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d6d18babe2d41309b20280399c71d5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:10:28.797734 11244 mvcc.cc:205] Tried to move back new op lower bound from 7332738780289642496 to 7332738780121612288. Current Snapshot: MvccSnapshot[applied={T|T < 7332738780289642496}]
I20260924 03:10:28.803390 11245 mvcc.cc:205] Tried to move back new op lower bound from 7332738780289642496 to 7332738780121612288. Current Snapshot: MvccSnapshot[applied={T|T < 7332738780289642496}]
I20260924 03:10:28.815555 10753 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:10:28.815838 10753 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:10:28.819885 10753 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8035f8517c89495a94900b92183fa41e: Sending DeleteTablet for 3 replicas of tablet 8f21db3892b04ef8810ef4aac5acb6c4
I20260924 03:10:28.821075 11146 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f21db3892b04ef8810ef4aac5acb6c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:10:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59636
I20260924 03:10:28.821100 11013 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f21db3892b04ef8810ef4aac5acb6c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:10:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52048
I20260924 03:10:28.821672 10879 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f21db3892b04ef8810ef4aac5acb6c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:10:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36174
I20260924 03:10:28.829908 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 10816
I20260924 03:10:28.841001 11254 tablet_replica.cc:330] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b: stopping tablet replica
I20260924 03:10:28.841563 11254 raft_consensus.cc:2229] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:10:28.842170 11254 raft_consensus.cc:2258] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:10:28.843014 11255 tablet_replica.cc:330] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879: stopping tablet replica
I20260924 03:10:28.843416 11255 raft_consensus.cc:2229] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:10:28.843914 11255 raft_consensus.cc:2258] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:10:28.844344 11256 tablet_replica.cc:330] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a: stopping tablet replica
I20260924 03:10:28.846171 11256 raft_consensus.cc:2229] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:10:28.846455 11255 ts_tablet_manager.cc:1905] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:10:28.847069 11256 raft_consensus.cc:2258] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:10:28.849824 11256 ts_tablet_manager.cc:1905] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 03:10:28 dist-test-slave-5l98 krb5kdc[10710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:10:28.854743 11255 ts_tablet_manager.cc:1918] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:10:28.854933 11255 log.cc:1196] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/8f21db3892b04ef8810ef4aac5acb6c4
I20260924 03:10:28.855545 11255 ts_tablet_manager.cc:1939] T 8f21db3892b04ef8810ef4aac5acb6c4 P 76858f8df2ab47938cbb4a4682abd879: Deleting consensus metadata
I20260924 03:10:28.858724 11254 ts_tablet_manager.cc:1905] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:10:28.859042 11256 ts_tablet_manager.cc:1918] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:10:28.859292 11256 log.cc:1196] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/8f21db3892b04ef8810ef4aac5acb6c4
I20260924 03:10:28.859912 11256 ts_tablet_manager.cc:1939] T 8f21db3892b04ef8810ef4aac5acb6c4 P 4989246e85904134b7ed1a99da335c7a: Deleting consensus metadata
I20260924 03:10:28.860630 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:28.831737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36186 (local address 127.0.97.1:44915)
0924 03:10:28.831974 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:28.831983 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:28.835274 (+ 3291us) server_negotiation.cc:407] Connection header received
0924 03:10:28.835507 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:28.835515 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:28.835628 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:28.835777 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:28.842880 (+ 7103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.844421 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.845830 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.846579 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.846715 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:28.852353 (+ 5638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:28.852368 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:28.852392 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:28.852441 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:28.856782 (+ 4341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:28.857847 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:28.857857 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:28.857861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:28.858050 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:28.858516 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:28.858520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:28.858524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:28.859004 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:28.859338 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:28.859986 (+ 648us) server_negotiation.cc:299] Negotiation successful
0924 03:10:28.860250 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 03:10:28.862133 10939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:10:28.862452 10741 catalog_manager.cc:4841] TS 4989246e85904134b7ed1a99da335c7a (127.0.97.1:44915): tablet 8f21db3892b04ef8810ef4aac5acb6c4 (table test-table [id=0b59c13dd0934d09b588d636f2ecff67]) successfully deleted
I20260924 03:10:28.863808 10741 catalog_manager.cc:4841] TS 76858f8df2ab47938cbb4a4682abd879 (127.0.97.2:33433): tablet 8f21db3892b04ef8810ef4aac5acb6c4 (table test-table [id=0b59c13dd0934d09b588d636f2ecff67]) successfully deleted
I20260924 03:10:28.866941 11254 ts_tablet_manager.cc:1918] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:10:28.867122 11254 log.cc:1196] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/8f21db3892b04ef8810ef4aac5acb6c4
I20260924 03:10:28.867806 11254 ts_tablet_manager.cc:1939] T 8f21db3892b04ef8810ef4aac5acb6c4 P 7d6d18babe2d41309b20280399c71d5b: Deleting consensus metadata
I20260924 03:10:28.869868 10739 catalog_manager.cc:4841] TS 7d6d18babe2d41309b20280399c71d5b (127.0.97.3:33413): tablet 8f21db3892b04ef8810ef4aac5acb6c4 (table test-table [id=0b59c13dd0934d09b588d636f2ecff67]) successfully deleted
I20260924 03:10:28.967000 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 10816
I20260924 03:10:28.982831 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 10950
Sep 24 03:10:28 dist-test-slave-5l98 krb5kdc[10710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 03:10:29.008031 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:28.983767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52060 (local address 127.0.97.2:33433)
0924 03:10:28.983929 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:28.983933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:28.983951 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:10:28.984098 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:28.984103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:28.984186 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:28.984273 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:28.985615 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.986441 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.987712 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:28.988121 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:28.988227 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:29.000888 (+ 12661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:29.000908 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:29.000933 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:29.001007 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:29.004938 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:29.005730 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:29.005738 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:29.005741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:29.005907 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:29.006417 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:29.006421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:29.006423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:29.006815 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:29.007038 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:29.007446 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 03:10:29.007610 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 03:10:29.008936 11073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:10:29.100569 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 10950
I20260924 03:10:29.118029 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 11083
Sep 24 03:10:29 dist-test-slave-5l98 krb5kdc[10710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 03:10:29.133608 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:29.118881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59644 (local address 127.0.97.3:33413)
0924 03:10:29.119063 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:29.119068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:29.119282 (+ 214us) server_negotiation.cc:407] Connection header received
0924 03:10:29.119412 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:29.119416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:29.119483 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:29.119573 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:29.121016 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:29.121819 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:29.123085 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:29.123457 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:29.123552 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:29.127871 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:29.127885 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:29.127908 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:29.127949 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:29.130792 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:29.131468 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:29.131475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:29.131477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:29.131622 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:29.131965 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:29.131969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:29.131971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:29.132373 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:29.132713 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:29.133030 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 03:10:29.133275 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:10:29.134256 11206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:10:29.224210 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 11083
I20260924 03:10:29.241060 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 10723
I20260924 03:10:29.242281 10785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:10:29.351703 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 10723
2026-09-24T03:10:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4509 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:10:29 dist-test-slave-5l98 krb5kdc[11264](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:10:29 dist-test-slave-5l98 krb5kdc[11264](info): set up 2 sockets
Sep 24 03:10:29 dist-test-slave-5l98 krb5kdc[11264](info): commencing operation
krb5kdc: starting...
W20260924 03:10:30.509610 388 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:10:30 dist-test-slave-5l98 krb5kdc[11264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:10:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:10:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260924 03:10:30.861824 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskYGuLdJ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/postgres -l logfile start
2026-09-24 03:10:33.871 UTC [11294] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 03:10:33.872 UTC [11294] LOG: listening on IPv4 address "127.0.97.20", port 46067
2026-09-24 03:10:33.877 UTC [11294] LOG: listening on Unix socket "/tmp/.s.PGSQL.46067"
2026-09-24 03:10:33.903 UTC [11297] LOG: database system was shut down at 2026-09-24 03:10:31 UTC
2026-09-24 03:10:33.908 UTC [11294] LOG: database system is ready to accept connections
I20260924 03:10:34.881227 388 mini_postgres.cc:96] Postgres bound to 46067
2026-09-24 03:10:34.899 UTC [11305] FATAL: database "slave" does not exist
127.0.97.20:46067 - accepting connections
I20260924 03:10:34.900306 388 mini_ranger.cc:161] Starting Ranger...
I20260924 03:10:34.928525 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 03:10:35.579697 388 mini_ranger.cc:87] Created ranger Postgres database
I20260924 03:10:35.579813 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ranger-admin
2026-09-24 03:10:35,900 [I] DB FLAVOR :POSTGRES
2026-09-24 03:10:35,900 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 03:10:35,900 [I] Checking connection..
2026-09-24 03:10:35,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 03:10:36,163 [I] Checking connection passed.
2026-09-24 03:10:36,163 [I] --------- Verifying version history table ---------
2026-09-24 03:10:36,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:10:36,424 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 03:10:36,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:10:36,691 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 03:10:36,692 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 03:10:36,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 03:10:36.960 UTC [11369] WARNING: there is no transaction in progress
2026-09-24 03:10:36,967 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 03:10:36,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:10:37,237 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 03:10:37,237 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 03:10:37,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 03:10:37,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 03:10:37,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5l98.c.gcp-upstream.internal','N') ;"
2026-09-24 03:10:38,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 03:10:38,276 [I] Table x_portal_user does not exist in database ranger
2026-09-24 03:10:38,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 03:10:38,544 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 03:10:38,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 03:10:38,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 03:10:39,076 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 03:10:39,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 03:10:40.254 UTC [11477] WARNING: there is no transaction in progress
2026-09-24 03:10:40.275 UTC [11477] WARNING: there is no transaction in progress
2026-09-24 03:10:40.299 UTC [11477] WARNING: there is no transaction in progress
2026-09-24 03:10:40.319 UTC [11477] WARNING: there is no transaction in progress
2026-09-24 03:10:40.339 UTC [11477] WARNING: there is no transaction in progress
2026-09-24 03:10:40.364 UTC [11477] WARNING: there is no transaction in progress
2026-09-24 03:10:40.438 UTC [11477] WARNING: there is no transaction in progress
2026-09-24 03:10:40.447 UTC [11477] WARNING: there is no transaction in progress
2026-09-24 03:10:40.469 UTC [11477] WARNING: there is no transaction in progress
2026-09-24 03:10:40.482 UTC [11477] WARNING: there is no transaction in progress
2026-09-24 03:10:40.496 UTC [11477] WARNING: there is no transaction in progress
2026-09-24 03:10:40.513 UTC [11477] WARNING: there is no transaction in progress
2026-09-24 03:10:40.525 UTC [11477] WARNING: there is no transaction in progress
2026-09-24 03:10:40.534 UTC [11477] WARNING: there is no transaction in progress
2026-09-24 03:10:40.543 UTC [11477] WARNING: there is no transaction in progress
2026-09-24 03:10:40.552 UTC [11477] WARNING: there is no transaction in progress
2026-09-24 03:10:41,422 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 03:10:41,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 03:10:41,675 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 03:10:41,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 03:10:41,931 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 03:10:41,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 03:10:42,203 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 03:10:42,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 03:10:42,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 03:10:42,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5l98.c.gcp-upstream.internal';"
2026-09-24 03:10:43,009 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 03:10:43,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 03:10:43,250 [I] DB_PATCHES have already been applied
I20260924 03:10:43.256273 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskYGuLdJ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 03:10:43.256367 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260924 03:10:43.271656 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:38337
Sep 24, 2026 3:10:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:10:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:10:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 3:10:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 3:10:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 3:10:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 3:10:44 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 03:10:44.427753 388 mini_ranger.cc:160] Time spent starting Ranger: real 9.528s user 0.008s sys 0.032s
Sep 24, 2026 3:11:05 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 3:11:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 3:11:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 3:11:05 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 3:11:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 3:11:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 3:11:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 3:11:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 3:11:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 3:11:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 3:11:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 3:11:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 3:11:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 3:11:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 3:11:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 3:11:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 3:11:05 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 3:11:06 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 03:11:07.036782 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 03:11:07 dist-test-slave-5l98 krb5kdc[11264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219467, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 03:11:07.221745 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 03:11:07.243042 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 03:11:08.367503 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 03:11:08.367689 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ranger-kms
I20260924 03:11:08.367825 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:46067
2026-09-24 03:11:08,780 [I] DB FLAVOR :POSTGRES
2026-09-24 03:11:08,780 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 03:11:08,780 [I] Checking connection
2026-09-24 03:11:08,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 03:11:09,029 [I] connection success
2026-09-24 03:11:09,029 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 03:11:09,029 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 03:11:09,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 03:11:09,279 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 03:11:09,279 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 03:11:09,279 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 03:11:09,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46067/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 03:11:09,574 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 03:11:10.133926 388 mini_ranger_kms.cc:310] Created kms service
I20260924 03:11:10.358570 388 mini_ranger_kms.cc:326] Created kudu user
I20260924 03:11:10.407474 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 03:11:10 dist-test-slave-5l98 krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219467, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 03:11:10.858568 388 mini_ranger_kms.cc:384] Added ranger policy
I20260924 03:11:10.858969 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 03:11:10.859036 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260924 03:11:10.877183 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11658
I20260924 03:11:10.877404 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:40087
Sep 24, 2026 3:11:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:11:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:11:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 3:11:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 3:11:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 3:11:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 03:11:13.239467 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.018s user 0.115s sys 0.311s
I20260924 03:11:13.239717 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 03:11:13.239929 388 mini_ranger_kms.cc:401] 127.0.97.20:40087/kms/v1/keys
Sep 24 03:11:13 dist-test-slave-5l98 krb5kdc[11264](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790219473, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 03:11:13 dist-test-slave-5l98 krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219473, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 24 03:11:14 dist-test-slave-5l98 krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219473, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 24 03:11:16 dist-test-slave-5l98 krb5kdc[11264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:11:16.510903 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39899
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39899
--ranger_config_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40087/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:11:16.875530 11708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:11:16.875913 11708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:11:16.876070 11708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:11:16.883771 11708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:11:16.883947 11708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:11:16.884016 11708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:11:16.884089 11708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:11:16.884215 11708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:11:16.895623 11708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:40087/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39899
--ranger_config_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:11:16.897096 11708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:11:16.898494 11708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:11:16.911947 11714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:16.912061 11713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:16.912726 11716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:16.914719 11708 server_base.cc:1034] running on GCE node
I20260924 03:11:16.915723 11708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:11:16.917667 11708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:11:16.918931 11708 hybrid_clock.cc:648] HybridClock initialized: now 1790219476918886 us; error 42 us; skew 500 ppm
I20260924 03:11:16.919329 11708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:11:16 dist-test-slave-5l98 krb5kdc[11264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:11:16.939251 11708 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:11:16.941780 11708 webserver.cc:460] Webserver started at http://127.0.97.62:45557/ using document root <none> and password file <none>
I20260924 03:11:16.942585 11708 fs_manager.cc:359] Metadata directory not provided
I20260924 03:11:16.942742 11708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:11:16.943084 11708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:11:16 dist-test-slave-5l98 krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 03:11:17.011099 11708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "beee2c33543945408ade8cbab792c3e8"
format_stamp: "Formatted at 2026-09-24 03:11:17 on dist-test-slave-5l98"
server_key: "859dbe7bbe6d5d76b4e908931a715ad4"
server_key_iv: "6c4f28b73739cf14ea627ce9c50822cc"
server_key_version: "kuduclusterkey@0"
I20260924 03:11:17.011902 11708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "beee2c33543945408ade8cbab792c3e8"
format_stamp: "Formatted at 2026-09-24 03:11:17 on dist-test-slave-5l98"
server_key: "859dbe7bbe6d5d76b4e908931a715ad4"
server_key_iv: "6c4f28b73739cf14ea627ce9c50822cc"
server_key_version: "kuduclusterkey@0"
I20260924 03:11:17.017484 11708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 03:11:17.038648 11723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:17.040428 11708 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260924 03:11:17.040684 11708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "beee2c33543945408ade8cbab792c3e8"
format_stamp: "Formatted at 2026-09-24 03:11:17 on dist-test-slave-5l98"
server_key: "859dbe7bbe6d5d76b4e908931a715ad4"
server_key_iv: "6c4f28b73739cf14ea627ce9c50822cc"
server_key_version: "kuduclusterkey@0"
I20260924 03:11:17.040943 11708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:11:17.105405 11708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:11:17.111581 11708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:11:17.112037 11708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:11:17.134686 11708 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39899
I20260924 03:11:17.134688 11774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39899 every 8 connection(s)
I20260924 03:11:17.136195 11708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:11:17.140317 11775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:11:17.144754 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 11708
I20260924 03:11:17.144948 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:11:17.145680 388 external_mini_cluster.cc:1428] Setting key afb794519447775c9ec322b9305b70fe
I20260924 03:11:17.157881 11775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8: Bootstrap starting.
I20260924 03:11:17.161593 11775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8: Neither blocks nor log segments found. Creating new log.
I20260924 03:11:17.163183 11775 log.cc:824] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8: Log is configured to *not* fsync() on all Append() calls
I20260924 03:11:17.167068 11775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8: No bootstrap required, opened a new log
Sep 24 03:11:17 dist-test-slave-5l98 krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:11:17.179234 11775 raft_consensus.cc:348] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beee2c33543945408ade8cbab792c3e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39899 } }
I20260924 03:11:17.179677 11775 raft_consensus.cc:374] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:11:17.179806 11775 raft_consensus.cc:729] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beee2c33543945408ade8cbab792c3e8, State: Initialized, Role: FOLLOWER
I20260924 03:11:17.180622 11775 consensus_queue.cc:260] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beee2c33543945408ade8cbab792c3e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39899 } }
I20260924 03:11:17.180894 11775 raft_consensus.cc:388] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:11:17.181071 11775 raft_consensus.cc:482] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:11:17.181324 11775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:11:17.184131 11775 raft_consensus.cc:504] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beee2c33543945408ade8cbab792c3e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39899 } }
I20260924 03:11:17.184728 11775 leader_election.cc:302] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: beee2c33543945408ade8cbab792c3e8; no voters:
I20260924 03:11:17.185387 11775 leader_election.cc:288] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:11:17.185506 11780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:11:17.188782 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:17.147781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.0.97.62:39899)
0924 03:11:17.161107 (+ 13326us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:17.161127 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:17.161182 (+ 55us) server_negotiation.cc:407] Connection header received
0924 03:11:17.164126 (+ 2944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:17.164151 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:17.164680 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:17.167060 (+ 2380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:11:17.167639 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:17.170078 (+ 2439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:17.172343 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:17.173682 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:17.173863 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:17.179021 (+ 5158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:11:17.179032 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:11:17.179054 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:11:17.179102 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:11:17.182882 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:17.183765 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:17.183774 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:17.183782 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:17.183986 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:17.184445 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:17.184450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:17.184453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:17.185222 (+ 769us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:11:17.186613 (+ 1391us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:17.186914 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 03:11:17.187347 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13083,"thread_start_us":12827,"threads_started":1}
I20260924 03:11:17.189872 11780 raft_consensus.cc:686] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8 [term 1 LEADER]: Becoming Leader. State: Replica: beee2c33543945408ade8cbab792c3e8, State: Running, Role: LEADER
I20260924 03:11:17.190686 11780 consensus_queue.cc:237] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beee2c33543945408ade8cbab792c3e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39899 } }
I20260924 03:11:17.195700 11775 sys_catalog.cc:564] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:11:17.196774 11782 sys_catalog.cc:455] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "beee2c33543945408ade8cbab792c3e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beee2c33543945408ade8cbab792c3e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39899 } } }
I20260924 03:11:17.197077 11782 sys_catalog.cc:458] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8 [sys.catalog]: This master's current role is: LEADER
I20260924 03:11:17.196242 11781 sys_catalog.cc:455] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader beee2c33543945408ade8cbab792c3e8. Latest consensus state: current_term: 1 leader_uuid: "beee2c33543945408ade8cbab792c3e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beee2c33543945408ade8cbab792c3e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39899 } } }
I20260924 03:11:17.197772 11781 sys_catalog.cc:458] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8 [sys.catalog]: This master's current role is: LEADER
I20260924 03:11:17.207111 11785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:11:17.216959 11785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:11:17.239514 11785 catalog_manager.cc:1347] Generated new cluster ID: e099caee928e4f45829794bfa821cbbb
I20260924 03:11:17.239647 11785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:11:17.253602 11785 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:11:17.255481 11785 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:11:17.265868 11785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8: Generated new TSK 0
I20260924 03:11:17.266942 11785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:11:17.269304 11775 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 03:11:18.185384 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40087/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39899
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:11:18 dist-test-slave-5l98 krb5kdc[11264](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790219478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 03:11:18.623952 11820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:11:18.624384 11820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:11:18.624635 11820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:11:18.636608 11820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:11:18.636755 11820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:11:18.637445 11820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:11:18.653765 11820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:40087/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:11:18.655669 11820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:11:18.657425 11820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:11:18.675591 11830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:18.677135 11831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:18.688898 11833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:18.689705 11820 server_base.cc:1034] running on GCE node
I20260924 03:11:18.690516 11820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:11:18.691979 11820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:11:18.693245 11820 hybrid_clock.cc:648] HybridClock initialized: now 1790219478693188 us; error 57 us; skew 500 ppm
I20260924 03:11:18.693619 11820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:11:18 dist-test-slave-5l98 krb5kdc[11264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:11:18.697675 11820 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:11:18.699787 11820 webserver.cc:460] Webserver started at http://127.0.97.1:34713/ using document root <none> and password file <none>
I20260924 03:11:18.700822 11820 fs_manager.cc:359] Metadata directory not provided
I20260924 03:11:18.700977 11820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:11:18.701398 11820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:11:18 dist-test-slave-5l98 krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 03:11:18.721913 11820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "797e8b0107f84c8da560f23384e56587"
format_stamp: "Formatted at 2026-09-24 03:11:18 on dist-test-slave-5l98"
server_key: "3f22a79a06ec4149b481f7e627faa0dc"
server_key_iv: "123eae9f8718bf5e9d62fef07e664f25"
server_key_version: "kuduclusterkey@0"
I20260924 03:11:18.723240 11820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "797e8b0107f84c8da560f23384e56587"
format_stamp: "Formatted at 2026-09-24 03:11:18 on dist-test-slave-5l98"
server_key: "3f22a79a06ec4149b481f7e627faa0dc"
server_key_iv: "123eae9f8718bf5e9d62fef07e664f25"
server_key_version: "kuduclusterkey@0"
I20260924 03:11:18.748122 11820 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.015s sys 0.007s
I20260924 03:11:18.771831 11840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:18.773789 11820 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:11:18.774055 11820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "797e8b0107f84c8da560f23384e56587"
format_stamp: "Formatted at 2026-09-24 03:11:18 on dist-test-slave-5l98"
server_key: "3f22a79a06ec4149b481f7e627faa0dc"
server_key_iv: "123eae9f8718bf5e9d62fef07e664f25"
server_key_version: "kuduclusterkey@0"
I20260924 03:11:18.774417 11820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:11:18.835152 11820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:11:18.840605 11820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:11:18.841045 11820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:11:18.842357 11820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:11:18.844626 11820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:11:18.844727 11820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:18.844887 11820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:11:18.844951 11820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:11:18 dist-test-slave-5l98 krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 03:11:18.992003 11820 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33075
I20260924 03:11:18.992400 11952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33075 every 8 connection(s)
I20260924 03:11:18.993767 11820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:11:19.000445 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 11820
I20260924 03:11:19.000679 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:11:19.001291 388 external_mini_cluster.cc:1428] Setting key 15088db02cc66b639eabddcc0dd08af6
Sep 24 03:11:19 dist-test-slave-5l98 krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:11:19.028314 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:18.998572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39899 (local address 127.0.97.1:36351)
0924 03:11:19.000111 (+ 1539us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:19.000168 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:19.002727 (+ 2559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:19.003768 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:19.003779 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:19.004364 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:11:19.005171 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:19.005201 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:19.006679 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:19.006686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:19.008850 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:19.008865 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:19.010595 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:19.010604 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:19.010940 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:19.011850 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:11:19.011886 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:11:19.018114 (+ 6228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:11:19.023040 (+ 4926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:19.023051 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:19.023084 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:19.023781 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:19.024323 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:19.024327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:19.024331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:19.024663 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:19.025433 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:11:19.025443 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:11:19.025921 (+ 478us) client_negotiation.cc:769] Sending connection context
0924 03:11:19.026272 (+ 351us) client_negotiation.cc:240] Negotiation successful
0924 03:11:19.026851 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":367,"threads_started":1}
I20260924 03:11:19.028751 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:18.998031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36351 (local address 127.0.97.62:39899)
0924 03:11:18.999567 (+ 1536us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:18.999573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:19.002269 (+ 2696us) server_negotiation.cc:407] Connection header received
0924 03:11:19.003163 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:19.003169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:19.003263 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:19.003360 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:11:19.005371 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:19.006461 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:19.009012 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:19.009956 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:19.010280 (+ 324us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:19.018500 (+ 8220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:11:19.018508 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:11:19.018514 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:11:19.019030 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:11:19.022808 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:19.023957 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:19.023964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:19.023967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:19.024121 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:19.024813 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:19.024819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:19.024823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:19.025205 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:11:19.025346 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:19.028153 (+ 2807us) server_negotiation.cc:299] Negotiation successful
0924 03:11:19.028382 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1410,"thread_start_us":253,"threads_started":1}
I20260924 03:11:19.030946 11953 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39899
I20260924 03:11:19.031303 11953 heartbeater.cc:464] Registering TS with master...
I20260924 03:11:19.032230 11953 heartbeater.cc:511] Master 127.0.97.62:39899 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 03:11:19.034643 11740 ts_manager.cc:194] Registered new tserver with Master: 797e8b0107f84c8da560f23384e56587 (127.0.97.1:33075)
I20260924 03:11:19.036276 11740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 03:11:19.069795 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40087/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39899
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:11:19 dist-test-slave-5l98 krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260924 03:11:19.539884 11959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:11:19.540385 11959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:11:19.540619 11959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:11:19.552341 11959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:11:19.552626 11959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:11:19.552881 11959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:11:19.569487 11959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:40087/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:11:19.571616 11959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:11:19.573542 11959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:11:19.595352 11966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:19.596112 11967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:19.608507 11969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:19.610683 11959 server_base.cc:1034] running on GCE node
I20260924 03:11:19.611550 11959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:11:19.612972 11959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:11:19.614292 11959 hybrid_clock.cc:648] HybridClock initialized: now 1790219479614188 us; error 90 us; skew 500 ppm
I20260924 03:11:19.614689 11959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:11:19 dist-test-slave-5l98 krb5kdc[11264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:11:19.618960 11959 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:11:19.621552 11959 webserver.cc:460] Webserver started at http://127.0.97.2:45065/ using document root <none> and password file <none>
I20260924 03:11:19.622480 11959 fs_manager.cc:359] Metadata directory not provided
I20260924 03:11:19.622617 11959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:11:19.622973 11959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:11:19 dist-test-slave-5l98 krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 03:11:19.644838 11959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "058d6fc795d5433f9d7455037f87cc1a"
format_stamp: "Formatted at 2026-09-24 03:11:19 on dist-test-slave-5l98"
server_key: "d9c23991cc75f08a9ccd641086987066"
server_key_iv: "65547f65e32bb103164e2cd40dadfea9"
server_key_version: "kuduclusterkey@0"
I20260924 03:11:19.646301 11959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "058d6fc795d5433f9d7455037f87cc1a"
format_stamp: "Formatted at 2026-09-24 03:11:19 on dist-test-slave-5l98"
server_key: "d9c23991cc75f08a9ccd641086987066"
server_key_iv: "65547f65e32bb103164e2cd40dadfea9"
server_key_version: "kuduclusterkey@0"
I20260924 03:11:19.666755 11959 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.007s sys 0.012s
I20260924 03:11:19.692387 11979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:19.694479 11959 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.001s
I20260924 03:11:19.694757 11959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "058d6fc795d5433f9d7455037f87cc1a"
format_stamp: "Formatted at 2026-09-24 03:11:19 on dist-test-slave-5l98"
server_key: "d9c23991cc75f08a9ccd641086987066"
server_key_iv: "65547f65e32bb103164e2cd40dadfea9"
server_key_version: "kuduclusterkey@0"
I20260924 03:11:19.695097 11959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:11:19.738762 11959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:11:19.744037 11959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:11:19.744422 11959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:11:19.745810 11959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:11:19.747673 11959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:11:19.747814 11959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260924 03:11:19.747967 11959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:11:19.748054 11959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:19.796178 12095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37521 every 8 connection(s)
I20260924 03:11:19.796180 11959 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37521
I20260924 03:11:19.797963 11959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:11:19.802835 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 11959
I20260924 03:11:19.803062 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:11:19.803750 388 external_mini_cluster.cc:1428] Setting key f3e813bbe65fdaa0b6e74e3aacb25a4c
Sep 24 03:11:19 dist-test-slave-5l98 krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:11:19.824030 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:19.800964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42963 (local address 127.0.97.62:39899)
0924 03:11:19.809004 (+ 8040us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:19.809009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:19.809026 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:11:19.809087 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:19.809091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:19.809166 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:19.809319 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:11:19.810904 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:19.811673 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:19.813571 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:19.813979 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:19.814119 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:19.817746 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:11:19.817753 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:11:19.817759 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:11:19.817811 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:11:19.820789 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:19.821563 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:19.821569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:19.821574 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:19.821751 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:19.822222 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:19.822227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:19.822231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:19.822519 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:11:19.822693 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:19.823412 (+ 719us) server_negotiation.cc:299] Negotiation successful
0924 03:11:19.823634 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7920,"thread_start_us":7797,"threads_started":1}
I20260924 03:11:19.825234 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:19.801810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39899 (local address 127.0.97.2:42963)
0924 03:11:19.802891 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:19.802950 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:19.805433 (+ 2483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:19.809603 (+ 4170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:19.809614 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:19.810102 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:11:19.810709 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:19.810726 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:19.811858 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:19.811863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:19.813416 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:19.813434 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:19.814227 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:19.814233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:19.814468 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:19.815198 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:11:19.815221 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:11:19.817503 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:11:19.820948 (+ 3445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:19.820958 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:19.820978 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:19.821427 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:19.821874 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:19.821878 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:19.821882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:19.822100 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:19.822664 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:11:19.822675 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:11:19.823035 (+ 360us) client_negotiation.cc:769] Sending connection context
0924 03:11:19.823307 (+ 272us) client_negotiation.cc:240] Negotiation successful
0924 03:11:19.823860 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":4468,"threads_started":1}
I20260924 03:11:19.826930 12096 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39899
I20260924 03:11:19.827271 12096 heartbeater.cc:464] Registering TS with master...
I20260924 03:11:19.828053 12096 heartbeater.cc:511] Master 127.0.97.62:39899 requested a full tablet report, sending...
I20260924 03:11:19.830189 11740 ts_manager.cc:194] Registered new tserver with Master: 058d6fc795d5433f9d7455037f87cc1a (127.0.97.2:37521)
I20260924 03:11:19.831040 11740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42963
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 03:11:19.874096 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40087/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39899
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:11:20.040619 11953 heartbeater.cc:503] Master 127.0.97.62:39899 was elected leader, sending a full tablet report...
W20260924 03:11:20.226200 12102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:11:20.226504 12102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:11:20.226662 12102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:11:20.234118 12102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:11:20.234237 12102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:11:20.234408 12102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:11:20.244987 12102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:40087/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:11:20.246366 12102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:11:20.247587 12102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:11:20.261526 12108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:20.261574 12107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:20.262059 12110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:20.262548 12102 server_base.cc:1034] running on GCE node
I20260924 03:11:20.263186 12102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:11:20.264140 12102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:11:20.265386 12102 hybrid_clock.cc:648] HybridClock initialized: now 1790219480265323 us; error 51 us; skew 500 ppm
I20260924 03:11:20.265717 12102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:11:20 dist-test-slave-5l98 krb5kdc[11264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:11:20.273850 12102 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:11:20.275475 12102 webserver.cc:460] Webserver started at http://127.0.97.3:42851/ using document root <none> and password file <none>
I20260924 03:11:20.276175 12102 fs_manager.cc:359] Metadata directory not provided
I20260924 03:11:20.276326 12102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:11:20.276732 12102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:11:20 dist-test-slave-5l98 krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 03:11:20.291280 12102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "71f1f74eac5f498aaba663fa16742cfa"
format_stamp: "Formatted at 2026-09-24 03:11:20 on dist-test-slave-5l98"
server_key: "1467bfb842e5dc1aa6dc2fef38451f6a"
server_key_iv: "cb1cba0677ae70b613c3b0a687a90d4f"
server_key_version: "kuduclusterkey@0"
I20260924 03:11:20.292971 12102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "71f1f74eac5f498aaba663fa16742cfa"
format_stamp: "Formatted at 2026-09-24 03:11:20 on dist-test-slave-5l98"
server_key: "1467bfb842e5dc1aa6dc2fef38451f6a"
server_key_iv: "cb1cba0677ae70b613c3b0a687a90d4f"
server_key_version: "kuduclusterkey@0"
I20260924 03:11:20.298929 12102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260924 03:11:20.314368 12117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:20.315964 12102 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:11:20.316179 12102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "71f1f74eac5f498aaba663fa16742cfa"
format_stamp: "Formatted at 2026-09-24 03:11:20 on dist-test-slave-5l98"
server_key: "1467bfb842e5dc1aa6dc2fef38451f6a"
server_key_iv: "cb1cba0677ae70b613c3b0a687a90d4f"
server_key_version: "kuduclusterkey@0"
I20260924 03:11:20.316442 12102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:11:20.332971 12102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:11:20.337610 12102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:11:20.337934 12102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:11:20.338904 12102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:11:20.340257 12102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:11:20.340363 12102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:20.340559 12102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:11:20.340637 12102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 03:11:20.378648 12102 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45277
I20260924 03:11:20.378695 12229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45277 every 8 connection(s)
I20260924 03:11:20.379979 12102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:11:20.381628 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 12102
I20260924 03:11:20.381810 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:11:20.382242 388 external_mini_cluster.cc:1428] Setting key 3e4d959268cff6308cf605c5126f3540
Sep 24 03:11:20 dist-test-slave-5l98 krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:11:20.404397 12233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:20.384244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60457 (local address 127.0.97.62:39899)
0924 03:11:20.388955 (+ 4711us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:20.388959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:20.388977 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:11:20.389038 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:20.389042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:20.389170 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:20.389307 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:11:20.390849 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.391658 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.393378 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.393833 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.393939 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:20.398189 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:11:20.398194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:11:20.398197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:11:20.398239 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:11:20.401387 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:20.402227 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:20.402232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:20.402234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:20.402380 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:20.402818 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:20.402821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:20.402823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:20.403093 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:11:20.403234 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:20.403927 (+ 693us) server_negotiation.cc:299] Negotiation successful
0924 03:11:20.404110 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4595,"thread_start_us":4454,"threads_started":1}
I20260924 03:11:20.405444 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:20.383612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39899 (local address 127.0.97.3:60457)
0924 03:11:20.384578 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:20.384628 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:20.387101 (+ 2473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:20.389583 (+ 2482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:20.389593 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:20.390067 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:11:20.390660 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:20.390677 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.391810 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.391814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:20.393173 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:20.393185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.394068 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.394073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:20.394341 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:20.395101 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:11:20.395124 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:11:20.397917 (+ 2793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:11:20.401547 (+ 3630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:20.401559 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:20.401583 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:20.402056 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:20.402508 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:20.402511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:20.402514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:20.402696 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:20.403219 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:11:20.403225 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:11:20.403545 (+ 320us) client_negotiation.cc:769] Sending connection context
0924 03:11:20.403791 (+ 246us) client_negotiation.cc:240] Negotiation successful
0924 03:11:20.404195 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":388,"threads_started":1}
I20260924 03:11:20.407119 12230 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39899
I20260924 03:11:20.407460 12230 heartbeater.cc:464] Registering TS with master...
I20260924 03:11:20.408171 12230 heartbeater.cc:511] Master 127.0.97.62:39899 requested a full tablet report, sending...
I20260924 03:11:20.410040 11740 ts_manager.cc:194] Registered new tserver with Master: 71f1f74eac5f498aaba663fa16742cfa (127.0.97.3:45277)
I20260924 03:11:20.410980 11740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60457
I20260924 03:11:20.412102 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:11:20.431710 12233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:20.417924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35482 (local address 127.0.97.62:39899)
0924 03:11:20.418119 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:20.418124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:20.418670 (+ 546us) server_negotiation.cc:407] Connection header received
0924 03:11:20.418736 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:20.418740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:20.418823 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:20.418913 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:11:20.420633 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.421425 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.422973 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.423405 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.423515 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:20.425963 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:11:20.425968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:11:20.425971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:11:20.426016 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:11:20.428910 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:20.429629 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:20.429633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:20.429635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:20.429788 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:20.430279 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:20.430283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:20.430285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:20.430572 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:11:20.430763 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:20.431248 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 03:11:20.431437 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 03:11:20.437351 11739 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:11:20.445268 11739 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:11:20.481881 12243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:20.459733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33075 (local address 127.0.0.1:58230)
0924 03:11:20.466917 (+ 7184us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:20.466984 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:20.467289 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:20.468816 (+ 1527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:20.468827 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:20.468870 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:11:20.470039 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:20.470072 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.473089 (+ 3017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.473098 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:20.476526 (+ 3428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:20.476551 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.479717 (+ 3166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.479724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:20.480053 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:20.480130 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 03:11:20.480342 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 03:11:20.480755 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6828,"thread_start_us":6643,"threads_started":1}
I20260924 03:11:20.482497 12244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:20.460305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.0.97.1:33075)
0924 03:11:20.466282 (+ 5977us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:20.466294 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:20.467633 (+ 1339us) server_negotiation.cc:407] Connection header received
0924 03:11:20.467800 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:20.467813 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:20.468165 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:20.468405 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:11:20.470350 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.472855 (+ 2505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.476817 (+ 3962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.479024 (+ 2207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.479204 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:20.479529 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:20.481917 (+ 2388us) server_negotiation.cc:299] Negotiation successful
0924 03:11:20.482124 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5775,"thread_start_us":5610,"threads_started":1}
I20260924 03:11:20.484504 12245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:20.459733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37521 (local address 127.0.0.1:44850)
0924 03:11:20.469792 (+ 10059us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:20.469811 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:20.469911 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:20.470608 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:20.470616 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:20.470654 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:11:20.471191 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:20.471206 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.476517 (+ 5311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.476528 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:20.481535 (+ 5007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:20.481558 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.483601 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.483608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:20.483794 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:20.483831 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 03:11:20.483902 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 03:11:20.484064 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9896,"mutex_wait_us":756,"thread_start_us":8906,"threads_started":1}
I20260924 03:11:20.485387 12246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:20.460596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44850 (local address 127.0.97.2:37521)
0924 03:11:20.469929 (+ 9333us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:20.469940 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:20.469970 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:11:20.470074 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:20.470083 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:20.470352 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:20.471507 (+ 1155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:11:20.473390 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.476085 (+ 2695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.481818 (+ 5733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.483300 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.484750 (+ 1450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:20.484946 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:20.485005 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 03:11:20.485125 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9159,"thread_start_us":9042,"threads_started":1}
I20260924 03:11:20.487059 11888 tablet_service.cc:1467] Processing CreateTablet for tablet 7f5e098ab2d4488b87e1d0027f659558 (DEFAULT_TABLE table=test-table [id=cf249afe52d84569ae1dab5ecd7d68fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:11:20.488584 11888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f5e098ab2d4488b87e1d0027f659558. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:11:20.488983 12031 tablet_service.cc:1467] Processing CreateTablet for tablet 7f5e098ab2d4488b87e1d0027f659558 (DEFAULT_TABLE table=test-table [id=cf249afe52d84569ae1dab5ecd7d68fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:11:20.490073 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:20.469886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45277 (local address 127.0.0.1:46144)
0924 03:11:20.481892 (+ 12006us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:20.481915 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:20.482124 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:20.482866 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:20.482873 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:20.482909 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:11:20.483261 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:20.483272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.485414 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.485418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:20.487358 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:20.487373 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.489276 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.489282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:20.489448 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:20.489489 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 03:11:20.489592 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 03:11:20.489745 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11866,"thread_start_us":11706,"threads_started":1}
I20260924 03:11:20.490175 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:20.470224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.0.97.3:45277)
0924 03:11:20.481089 (+ 10865us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:20.481100 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:20.482010 (+ 910us) server_negotiation.cc:407] Connection header received
0924 03:11:20.482310 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:20.482321 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:20.482609 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:20.482806 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:11:20.483469 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.485217 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.487538 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.489046 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.489202 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:20.489424 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:20.489650 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 03:11:20.489828 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10657,"thread_start_us":10885,"threads_started":1}
I20260924 03:11:20.490569 12031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f5e098ab2d4488b87e1d0027f659558. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:11:20.494135 12165 tablet_service.cc:1467] Processing CreateTablet for tablet 7f5e098ab2d4488b87e1d0027f659558 (DEFAULT_TABLE table=test-table [id=cf249afe52d84569ae1dab5ecd7d68fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:11:20.495545 12165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f5e098ab2d4488b87e1d0027f659558. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:11:20.502475 12249 tablet_bootstrap.cc:492] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587: Bootstrap starting.
I20260924 03:11:20.505750 12249 tablet_bootstrap.cc:654] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587: Neither blocks nor log segments found. Creating new log.
I20260924 03:11:20.506589 12250 tablet_bootstrap.cc:492] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a: Bootstrap starting.
I20260924 03:11:20.507067 12249 log.cc:824] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587: Log is configured to *not* fsync() on all Append() calls
I20260924 03:11:20.510589 12249 tablet_bootstrap.cc:492] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587: No bootstrap required, opened a new log
I20260924 03:11:20.510661 12250 tablet_bootstrap.cc:654] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a: Neither blocks nor log segments found. Creating new log.
I20260924 03:11:20.510835 12252 tablet_bootstrap.cc:492] T 7f5e098ab2d4488b87e1d0027f659558 P 71f1f74eac5f498aaba663fa16742cfa: Bootstrap starting.
I20260924 03:11:20.510911 12249 ts_tablet_manager.cc:1397] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 03:11:20.512034 12250 log.cc:824] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:11:20.514217 12252 tablet_bootstrap.cc:654] T 7f5e098ab2d4488b87e1d0027f659558 P 71f1f74eac5f498aaba663fa16742cfa: Neither blocks nor log segments found. Creating new log.
I20260924 03:11:20.515270 12250 tablet_bootstrap.cc:492] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a: No bootstrap required, opened a new log
I20260924 03:11:20.515587 12250 ts_tablet_manager.cc:1397] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 03:11:20.515826 12252 log.cc:824] T 7f5e098ab2d4488b87e1d0027f659558 P 71f1f74eac5f498aaba663fa16742cfa: Log is configured to *not* fsync() on all Append() calls
I20260924 03:11:20.518290 12252 tablet_bootstrap.cc:492] T 7f5e098ab2d4488b87e1d0027f659558 P 71f1f74eac5f498aaba663fa16742cfa: No bootstrap required, opened a new log
I20260924 03:11:20.518553 12252 ts_tablet_manager.cc:1397] T 7f5e098ab2d4488b87e1d0027f659558 P 71f1f74eac5f498aaba663fa16742cfa: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 03:11:20.522698 12249 raft_consensus.cc:348] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058d6fc795d5433f9d7455037f87cc1a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 } } peers { permanent_uuid: "797e8b0107f84c8da560f23384e56587" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33075 } } peers { permanent_uuid: "71f1f74eac5f498aaba663fa16742cfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45277 } }
I20260924 03:11:20.523145 12249 raft_consensus.cc:374] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:11:20.523281 12249 raft_consensus.cc:729] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 797e8b0107f84c8da560f23384e56587, State: Initialized, Role: FOLLOWER
I20260924 03:11:20.523909 12249 consensus_queue.cc:260] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058d6fc795d5433f9d7455037f87cc1a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 } } peers { permanent_uuid: "797e8b0107f84c8da560f23384e56587" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33075 } } peers { permanent_uuid: "71f1f74eac5f498aaba663fa16742cfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45277 } }
I20260924 03:11:20.527048 12252 raft_consensus.cc:348] T 7f5e098ab2d4488b87e1d0027f659558 P 71f1f74eac5f498aaba663fa16742cfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058d6fc795d5433f9d7455037f87cc1a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 } } peers { permanent_uuid: "797e8b0107f84c8da560f23384e56587" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33075 } } peers { permanent_uuid: "71f1f74eac5f498aaba663fa16742cfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45277 } }
I20260924 03:11:20.527191 12250 raft_consensus.cc:348] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058d6fc795d5433f9d7455037f87cc1a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 } } peers { permanent_uuid: "797e8b0107f84c8da560f23384e56587" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33075 } } peers { permanent_uuid: "71f1f74eac5f498aaba663fa16742cfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45277 } }
I20260924 03:11:20.527689 12252 raft_consensus.cc:374] T 7f5e098ab2d4488b87e1d0027f659558 P 71f1f74eac5f498aaba663fa16742cfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:11:20.527794 12250 raft_consensus.cc:374] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:11:20.527870 12252 raft_consensus.cc:729] T 7f5e098ab2d4488b87e1d0027f659558 P 71f1f74eac5f498aaba663fa16742cfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71f1f74eac5f498aaba663fa16742cfa, State: Initialized, Role: FOLLOWER
I20260924 03:11:20.527973 12250 raft_consensus.cc:729] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058d6fc795d5433f9d7455037f87cc1a, State: Initialized, Role: FOLLOWER
I20260924 03:11:20.528867 12252 consensus_queue.cc:260] T 7f5e098ab2d4488b87e1d0027f659558 P 71f1f74eac5f498aaba663fa16742cfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058d6fc795d5433f9d7455037f87cc1a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 } } peers { permanent_uuid: "797e8b0107f84c8da560f23384e56587" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33075 } } peers { permanent_uuid: "71f1f74eac5f498aaba663fa16742cfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45277 } }
I20260924 03:11:20.528955 12250 consensus_queue.cc:260] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058d6fc795d5433f9d7455037f87cc1a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 } } peers { permanent_uuid: "797e8b0107f84c8da560f23384e56587" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33075 } } peers { permanent_uuid: "71f1f74eac5f498aaba663fa16742cfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45277 } }
I20260924 03:11:20.530386 12256 raft_consensus.cc:482] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:11:20.531172 12256 raft_consensus.cc:504] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058d6fc795d5433f9d7455037f87cc1a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 } } peers { permanent_uuid: "797e8b0107f84c8da560f23384e56587" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33075 } } peers { permanent_uuid: "71f1f74eac5f498aaba663fa16742cfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45277 } }
I20260924 03:11:20.532426 12249 ts_tablet_manager.cc:1428] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587: Time spent starting tablet: real 0.021s user 0.014s sys 0.004s
I20260924 03:11:20.534303 12230 heartbeater.cc:503] Master 127.0.97.62:39899 was elected leader, sending a full tablet report...
I20260924 03:11:20.534562 12250 ts_tablet_manager.cc:1428] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a: Time spent starting tablet: real 0.019s user 0.013s sys 0.004s
I20260924 03:11:20.535583 12096 heartbeater.cc:503] Master 127.0.97.62:39899 was elected leader, sending a full tablet report...
I20260924 03:11:20.538054 12252 ts_tablet_manager.cc:1428] T 7f5e098ab2d4488b87e1d0027f659558 P 71f1f74eac5f498aaba663fa16742cfa: Time spent starting tablet: real 0.019s user 0.013s sys 0.000s
I20260924 03:11:20.538568 12256 leader_election.cc:288] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 058d6fc795d5433f9d7455037f87cc1a (127.0.97.2:37521), 71f1f74eac5f498aaba663fa16742cfa (127.0.97.3:45277)
I20260924 03:11:20.547540 12246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:20.534221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36963 (local address 127.0.97.2:37521)
0924 03:11:20.536722 (+ 2501us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:20.536730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:20.536749 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:11:20.536822 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:20.536827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:20.536892 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:20.536994 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:11:20.541397 (+ 4403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.542427 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.544970 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.546244 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.546372 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:20.546467 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:20.547155 (+ 688us) server_negotiation.cc:299] Negotiation successful
0924 03:11:20.547286 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2348}
I20260924 03:11:20.547598 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:20.539149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54355 (local address 127.0.97.3:45277)
0924 03:11:20.539477 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:20.539483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:20.540472 (+ 989us) server_negotiation.cc:407] Connection header received
0924 03:11:20.540749 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:20.540754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:20.540822 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:20.540880 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:11:20.541695 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.542671 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.545012 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.546242 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.546372 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:20.546898 (+ 526us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:20.547155 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 03:11:20.547387 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260924 03:11:20.547730 12260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:20.538941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45277 (local address 127.0.97.1:54355)
0924 03:11:20.540377 (+ 1436us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:20.540395 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:20.540599 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:20.541099 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:20.541106 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:20.541135 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:11:20.541523 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:20.541537 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.542835 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.542839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:20.544857 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:20.544869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.546530 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.546541 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:20.546900 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:20.547008 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 03:11:20.547111 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 03:11:20.547253 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1300,"mutex_wait_us":79,"thread_start_us":1153,"threads_started":1}
I20260924 03:11:20.547928 12259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:20.533420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37521 (local address 127.0.97.1:36963)
0924 03:11:20.534339 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:20.534361 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:20.534515 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:20.537547 (+ 3032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:20.537553 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:20.537586 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:11:20.537980 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:20.537995 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.542785 (+ 4790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.542792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:20.544792 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:20.544805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.546569 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.546575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:20.546900 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:20.547006 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 03:11:20.547111 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 03:11:20.547322 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":110,"thread_start_us":3544,"threads_started":1}
I20260924 03:11:20.548789 12185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f5e098ab2d4488b87e1d0027f659558" candidate_uuid: "797e8b0107f84c8da560f23384e56587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f1f74eac5f498aaba663fa16742cfa" is_pre_election: true
I20260924 03:11:20.549137 12051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f5e098ab2d4488b87e1d0027f659558" candidate_uuid: "797e8b0107f84c8da560f23384e56587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058d6fc795d5433f9d7455037f87cc1a" is_pre_election: true
I20260924 03:11:20.549283 12185 raft_consensus.cc:2454] T 7f5e098ab2d4488b87e1d0027f659558 P 71f1f74eac5f498aaba663fa16742cfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 797e8b0107f84c8da560f23384e56587 in term 0.
I20260924 03:11:20.549580 12051 raft_consensus.cc:2454] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 797e8b0107f84c8da560f23384e56587 in term 0.
I20260924 03:11:20.550027 11842 leader_election.cc:302] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71f1f74eac5f498aaba663fa16742cfa, 797e8b0107f84c8da560f23384e56587; no voters:
W20260924 03:11:20.550114 12097 tablet.cc:2369] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:11:20.550485 12256 raft_consensus.cc:2781] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:11:20.550643 12256 raft_consensus.cc:482] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:11:20.550778 12256 raft_consensus.cc:3037] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:11:20.553071 12256 raft_consensus.cc:504] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058d6fc795d5433f9d7455037f87cc1a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 } } peers { permanent_uuid: "797e8b0107f84c8da560f23384e56587" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33075 } } peers { permanent_uuid: "71f1f74eac5f498aaba663fa16742cfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45277 } }
I20260924 03:11:20.553710 12256 leader_election.cc:288] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 [CANDIDATE]: Term 1 election: Requested vote from peers 058d6fc795d5433f9d7455037f87cc1a (127.0.97.2:37521), 71f1f74eac5f498aaba663fa16742cfa (127.0.97.3:45277)
I20260924 03:11:20.554344 12185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f5e098ab2d4488b87e1d0027f659558" candidate_uuid: "797e8b0107f84c8da560f23384e56587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f1f74eac5f498aaba663fa16742cfa"
I20260924 03:11:20.554252 12051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f5e098ab2d4488b87e1d0027f659558" candidate_uuid: "797e8b0107f84c8da560f23384e56587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058d6fc795d5433f9d7455037f87cc1a"
I20260924 03:11:20.554558 12051 raft_consensus.cc:3037] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:11:20.554558 12185 raft_consensus.cc:3037] T 7f5e098ab2d4488b87e1d0027f659558 P 71f1f74eac5f498aaba663fa16742cfa [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:11:20.557847 12051 raft_consensus.cc:2454] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 797e8b0107f84c8da560f23384e56587 in term 1.
I20260924 03:11:20.557854 12185 raft_consensus.cc:2454] T 7f5e098ab2d4488b87e1d0027f659558 P 71f1f74eac5f498aaba663fa16742cfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 797e8b0107f84c8da560f23384e56587 in term 1.
I20260924 03:11:20.558306 11842 leader_election.cc:302] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058d6fc795d5433f9d7455037f87cc1a, 797e8b0107f84c8da560f23384e56587; no voters:
I20260924 03:11:20.558609 12256 raft_consensus.cc:2781] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:11:20.558813 12256 raft_consensus.cc:686] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 [term 1 LEADER]: Becoming Leader. State: Replica: 797e8b0107f84c8da560f23384e56587, State: Running, Role: LEADER
I20260924 03:11:20.559378 12256 consensus_queue.cc:237] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058d6fc795d5433f9d7455037f87cc1a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 } } peers { permanent_uuid: "797e8b0107f84c8da560f23384e56587" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33075 } } peers { permanent_uuid: "71f1f74eac5f498aaba663fa16742cfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45277 } }
I20260924 03:11:20.563735 11738 catalog_manager.cc:5495] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 reported cstate change: term changed from 0 to 1, leader changed from <none> to 797e8b0107f84c8da560f23384e56587 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "797e8b0107f84c8da560f23384e56587" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058d6fc795d5433f9d7455037f87cc1a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "797e8b0107f84c8da560f23384e56587" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71f1f74eac5f498aaba663fa16742cfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45277 } health_report { overall_health: UNKNOWN } } }
I20260924 03:11:20.601433 12244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:20.595200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.0.97.1:33075)
0924 03:11:20.595404 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:20.595408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:20.595426 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:11:20.595516 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:20.595519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:20.595601 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:20.595742 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:11:20.596287 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.597259 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.598732 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.599205 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.599428 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:20.599973 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:11:20.600821 (+ 848us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:11:20.600953 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:20.601142 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 03:11:20.601223 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 03:11:20.610136 12185 raft_consensus.cc:1261] T 7f5e098ab2d4488b87e1d0027f659558 P 71f1f74eac5f498aaba663fa16742cfa [term 1 FOLLOWER]: Refusing update from remote peer 797e8b0107f84c8da560f23384e56587: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:11:20.610162 12051 raft_consensus.cc:1261] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a [term 1 FOLLOWER]: Refusing update from remote peer 797e8b0107f84c8da560f23384e56587: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:11:20.611212 12255 consensus_queue.cc:1035] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71f1f74eac5f498aaba663fa16742cfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:11:20.611573 12256 consensus_queue.cc:1035] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "058d6fc795d5433f9d7455037f87cc1a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:11:20.627102 12265 mvcc.cc:205] Tried to move back new op lower bound from 7332738992568209408 to 7332738992375152640. Current Snapshot: MvccSnapshot[applied={T|T < 7332738992568209408}]
I20260924 03:11:20.631191 12264 mvcc.cc:205] Tried to move back new op lower bound from 7332738992568209408 to 7332738992375152640. Current Snapshot: MvccSnapshot[applied={T|T < 7332738992568209408}]
I20260924 03:11:20.646714 11739 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:11:20.647045 11739 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:11:20.651490 11739 catalog_manager.cc:5782] T 00000000000000000000000000000000 P beee2c33543945408ade8cbab792c3e8: Sending DeleteTablet for 3 replicas of tablet 7f5e098ab2d4488b87e1d0027f659558
I20260924 03:11:20.652806 11888 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f5e098ab2d4488b87e1d0027f659558 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:11:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58230
I20260924 03:11:20.653103 12165 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f5e098ab2d4488b87e1d0027f659558 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:11:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46144
I20260924 03:11:20.653846 12031 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f5e098ab2d4488b87e1d0027f659558 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:11:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44850
I20260924 03:11:20.654992 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 11820
I20260924 03:11:20.671844 12273 tablet_replica.cc:330] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587: stopping tablet replica
I20260924 03:11:20.672433 12273 raft_consensus.cc:2229] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:11:20.673236 12273 raft_consensus.cc:2258] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:11:20.675592 12273 ts_tablet_manager.cc:1905] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:11:20.681556 12274 tablet_replica.cc:330] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a: stopping tablet replica
I20260924 03:11:20.682284 12275 tablet_replica.cc:330] T 7f5e098ab2d4488b87e1d0027f659558 P 71f1f74eac5f498aaba663fa16742cfa: stopping tablet replica
I20260924 03:11:20.683008 12275 raft_consensus.cc:2229] T 7f5e098ab2d4488b87e1d0027f659558 P 71f1f74eac5f498aaba663fa16742cfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:11:20.683565 12273 ts_tablet_manager.cc:1918] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 24 03:11:20 dist-test-slave-5l98 krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:11:20.683580 12274 raft_consensus.cc:2229] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:11:20.683779 12273 log.cc:1196] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/7f5e098ab2d4488b87e1d0027f659558
I20260924 03:11:20.684360 12274 raft_consensus.cc:2258] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:11:20.684383 12273 ts_tablet_manager.cc:1939] T 7f5e098ab2d4488b87e1d0027f659558 P 797e8b0107f84c8da560f23384e56587: Deleting consensus metadata
I20260924 03:11:20.684969 12275 raft_consensus.cc:2258] T 7f5e098ab2d4488b87e1d0027f659558 P 71f1f74eac5f498aaba663fa16742cfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:11:20.686630 11724 catalog_manager.cc:4841] TS 797e8b0107f84c8da560f23384e56587 (127.0.97.1:33075): tablet 7f5e098ab2d4488b87e1d0027f659558 (table test-table [id=cf249afe52d84569ae1dab5ecd7d68fb]) successfully deleted
I20260924 03:11:20.687680 12274 ts_tablet_manager.cc:1905] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:11:20.687979 12275 ts_tablet_manager.cc:1905] T 7f5e098ab2d4488b87e1d0027f659558 P 71f1f74eac5f498aaba663fa16742cfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:11:20.695312 12274 ts_tablet_manager.cc:1918] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:11:20.695536 12274 log.cc:1196] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/7f5e098ab2d4488b87e1d0027f659558
I20260924 03:11:20.695801 12275 ts_tablet_manager.cc:1918] T 7f5e098ab2d4488b87e1d0027f659558 P 71f1f74eac5f498aaba663fa16742cfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:11:20.696024 12244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:20.659142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.0.97.1:33075)
0924 03:11:20.659388 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:20.659398 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:20.667959 (+ 8561us) server_negotiation.cc:407] Connection header received
0924 03:11:20.668573 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:20.668584 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:20.668699 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:20.668794 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:11:20.671576 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.673371 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.676222 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.677269 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.677436 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:20.685090 (+ 7654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:11:20.685110 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:11:20.685129 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:11:20.685196 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:11:20.690985 (+ 5789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:20.692036 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:20.692045 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:20.692048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:20.692245 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:20.692831 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:20.692838 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:20.692842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:20.693390 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:11:20.693689 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:20.695145 (+ 1456us) server_negotiation.cc:299] Negotiation successful
0924 03:11:20.695540 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 03:11:20.696079 12274 ts_tablet_manager.cc:1939] T 7f5e098ab2d4488b87e1d0027f659558 P 058d6fc795d5433f9d7455037f87cc1a: Deleting consensus metadata
I20260924 03:11:20.696098 12275 log.cc:1196] T 7f5e098ab2d4488b87e1d0027f659558 P 71f1f74eac5f498aaba663fa16742cfa: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/7f5e098ab2d4488b87e1d0027f659558
I20260924 03:11:20.696761 12275 ts_tablet_manager.cc:1939] T 7f5e098ab2d4488b87e1d0027f659558 P 71f1f74eac5f498aaba663fa16742cfa: Deleting consensus metadata
I20260924 03:11:20.697211 11948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:11:20.698225 11725 catalog_manager.cc:4841] TS 058d6fc795d5433f9d7455037f87cc1a (127.0.97.2:37521): tablet 7f5e098ab2d4488b87e1d0027f659558 (table test-table [id=cf249afe52d84569ae1dab5ecd7d68fb]) successfully deleted
I20260924 03:11:20.699065 11725 catalog_manager.cc:4841] TS 71f1f74eac5f498aaba663fa16742cfa (127.0.97.3:45277): tablet 7f5e098ab2d4488b87e1d0027f659558 (table test-table [id=cf249afe52d84569ae1dab5ecd7d68fb]) successfully deleted
I20260924 03:11:20.797659 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 11820
I20260924 03:11:20.816011 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 11959
Sep 24 03:11:20 dist-test-slave-5l98 krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 03:11:20.832046 12246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:20.816951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44864 (local address 127.0.97.2:37521)
0924 03:11:20.817134 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:20.817142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:20.817165 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:11:20.817375 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:20.817379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:20.817454 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:20.817566 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:11:20.818779 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.819520 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.821033 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.821410 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.821569 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:20.826108 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:11:20.826199 (+ 91us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:11:20.826215 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:11:20.826252 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:11:20.829328 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:20.830058 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:20.830065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:20.830067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:20.830219 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:20.830607 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:20.830610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:20.830612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:20.830959 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:11:20.831163 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:20.831518 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 03:11:20.831754 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:11:20.832820 12091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:11:20.928750 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 11959
I20260924 03:11:20.945317 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 12102
Sep 24 03:11:20 dist-test-slave-5l98 krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 03:11:20.962775 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:20.946203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46150 (local address 127.0.97.3:45277)
0924 03:11:20.946420 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:20.946426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:20.946447 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:11:20.946603 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:20.946607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:20.946677 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:20.946778 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:11:20.948078 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.949007 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.950446 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:20.950923 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:20.951037 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:20.956130 (+ 5093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:11:20.956148 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:11:20.956164 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:11:20.956205 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:11:20.959400 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:20.960178 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:20.960195 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:20.960198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:20.960330 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:20.960880 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:20.960889 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:20.960893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:20.961401 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:11:20.961716 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:20.962136 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 03:11:20.962374 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 03:11:20.963480 12225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:11:21.055751 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 12102
I20260924 03:11:21.072645 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 11708
I20260924 03:11:21.073897 11770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:11:21.187808 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 11708
I20260924 03:11:21.209395 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 03:11:21.545564 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 03:11:21.545818 388 mini_ranger.cc:66] Stopping Ranger...
I20260924 03:11:21.892162 388 mini_ranger.cc:68] Stopped Ranger
2026-09-24 03:11:21.892 UTC [11294] LOG: received smart shutdown request
2026-09-24 03:11:21.897 UTC [11294] LOG: background worker "logical replication launcher" (PID 11303) exited with exit code 1
2026-09-24 03:11:21.897 UTC [11298] LOG: shutting down
2026-09-24 03:11:22.133 UTC [11294] LOG: database system is shut down
2026-09-24T03:11:22Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52904 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:11:22 dist-test-slave-5l98 krb5kdc[12289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:11:22 dist-test-slave-5l98 krb5kdc[12289](info): set up 2 sockets
Sep 24 03:11:22 dist-test-slave-5l98 krb5kdc[12289](info): commencing operation
krb5kdc: starting...
W20260924 03:11:23.372951 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:11:23 dist-test-slave-5l98 krb5kdc[12289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:11:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:11:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260924 03:11:23.716061 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskYGuLdJ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/postgres -l logfile start
2026-09-24 03:11:26.730 UTC [12319] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 03:11:26.730 UTC [12319] LOG: listening on IPv4 address "127.0.97.20", port 49513
2026-09-24 03:11:26.735 UTC [12319] LOG: listening on Unix socket "/tmp/.s.PGSQL.49513"
2026-09-24 03:11:26.758 UTC [12322] LOG: database system was shut down at 2026-09-24 03:11:24 UTC
2026-09-24 03:11:26.762 UTC [12319] LOG: database system is ready to accept connections
I20260924 03:11:27.752584 388 mini_postgres.cc:96] Postgres bound to 49513
2026-09-24 03:11:27.769 UTC [12330] FATAL: database "slave" does not exist
127.0.97.20:49513 - accepting connections
I20260924 03:11:27.770624 388 mini_ranger.cc:161] Starting Ranger...
I20260924 03:11:27.796748 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 03:11:28.435209 388 mini_ranger.cc:87] Created ranger Postgres database
I20260924 03:11:28.435326 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ranger-admin
2026-09-24 03:11:28,715 [I] DB FLAVOR :POSTGRES
2026-09-24 03:11:28,716 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 03:11:28,716 [I] Checking connection..
2026-09-24 03:11:28,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 03:11:28,962 [I] Checking connection passed.
2026-09-24 03:11:28,963 [I] --------- Verifying version history table ---------
2026-09-24 03:11:28,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:11:29,219 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 03:11:29,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:11:29,476 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 03:11:29,477 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 03:11:29,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 03:11:29.712 UTC [12394] WARNING: there is no transaction in progress
2026-09-24 03:11:29,719 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 03:11:29,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:11:29,963 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 03:11:29,963 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 03:11:29,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 03:11:30,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 03:11:30,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5l98.c.gcp-upstream.internal','N') ;"
2026-09-24 03:11:30,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 03:11:31,011 [I] Table x_portal_user does not exist in database ranger
2026-09-24 03:11:31,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 03:11:31,248 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 03:11:31,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 03:11:31,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 03:11:31,741 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 03:11:31,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 03:11:32.835 UTC [12502] WARNING: there is no transaction in progress
2026-09-24 03:11:32.856 UTC [12502] WARNING: there is no transaction in progress
2026-09-24 03:11:32.877 UTC [12502] WARNING: there is no transaction in progress
2026-09-24 03:11:32.896 UTC [12502] WARNING: there is no transaction in progress
2026-09-24 03:11:32.915 UTC [12502] WARNING: there is no transaction in progress
2026-09-24 03:11:32.936 UTC [12502] WARNING: there is no transaction in progress
2026-09-24 03:11:33.001 UTC [12502] WARNING: there is no transaction in progress
2026-09-24 03:11:33.010 UTC [12502] WARNING: there is no transaction in progress
2026-09-24 03:11:33.029 UTC [12502] WARNING: there is no transaction in progress
2026-09-24 03:11:33.042 UTC [12502] WARNING: there is no transaction in progress
2026-09-24 03:11:33.054 UTC [12502] WARNING: there is no transaction in progress
2026-09-24 03:11:33.071 UTC [12502] WARNING: there is no transaction in progress
2026-09-24 03:11:33.083 UTC [12502] WARNING: there is no transaction in progress
2026-09-24 03:11:33.090 UTC [12502] WARNING: there is no transaction in progress
2026-09-24 03:11:33.099 UTC [12502] WARNING: there is no transaction in progress
2026-09-24 03:11:33.108 UTC [12502] WARNING: there is no transaction in progress
2026-09-24 03:11:33,908 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 03:11:33,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 03:11:34,156 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 03:11:34,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 03:11:34,405 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 03:11:34,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 03:11:34,649 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 03:11:34,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 03:11:34,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 03:11:35,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5l98.c.gcp-upstream.internal';"
2026-09-24 03:11:35,392 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 03:11:35,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 03:11:35,667 [I] DB_PATCHES have already been applied
I20260924 03:11:35.673584 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskYGuLdJ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 03:11:35.673702 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260924 03:11:35.688186 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:60623
Sep 24, 2026 3:11:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:11:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:11:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 3:11:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 3:11:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 3:11:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 3:11:36 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 03:11:36.830240 388 mini_ranger.cc:160] Time spent starting Ranger: real 9.060s user 0.005s sys 0.035s
Sep 24, 2026 3:11:57 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 3:11:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 24, 2026 3:11:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 3:11:57 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 3:11:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 3:11:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 3:11:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 3:11:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 3:11:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 3:11:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 3:11:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 3:11:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 3:11:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 3:11:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 3:11:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 3:11:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 3:11:57 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 3:11:58 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 03:11:59.426867 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 03:11:59 dist-test-slave-5l98 krb5kdc[12289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219519, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 03:11:59.612532 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 03:11:59.634999 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 03:12:00.789162 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 03:12:00.789393 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ranger-kms
I20260924 03:12:00.789467 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:49513
2026-09-24 03:12:01,143 [I] DB FLAVOR :POSTGRES
2026-09-24 03:12:01,143 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 03:12:01,143 [I] Checking connection
2026-09-24 03:12:01,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 03:12:01,378 [I] connection success
2026-09-24 03:12:01,379 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 03:12:01,379 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 03:12:01,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 03:12:01,642 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 03:12:01,643 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 03:12:01,643 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 03:12:01,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49513/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 03:12:01,923 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 03:12:02.409598 388 mini_ranger_kms.cc:310] Created kms service
I20260924 03:12:02.660056 388 mini_ranger_kms.cc:326] Created kudu user
I20260924 03:12:02.708365 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 03:12:02 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219519, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 03:12:03.146571 388 mini_ranger_kms.cc:384] Added ranger policy
I20260924 03:12:03.146906 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskYGuLdJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskYGuLdJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 03:12:03.146981 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260924 03:12:03.161998 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12683
I20260924 03:12:03.162149 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:56307
Sep 24, 2026 3:12:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:12:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:12:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 3:12:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 3:12:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 3:12:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 03:12:05.458877 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.846s user 0.077s sys 0.294s
I20260924 03:12:05.459129 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 03:12:05.459306 388 mini_ranger_kms.cc:401] 127.0.97.20:56307/kms/v1/keys
Sep 24 03:12:05 dist-test-slave-5l98 krb5kdc[12289](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790219525, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 03:12:06 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219525, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 24 03:12:06 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219525, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 24 03:12:08 dist-test-slave-5l98 krb5kdc[12289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:12:08.745401 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42023
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42023
--ranger_config_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56307/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:09.086880 12733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:09.087159 12733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:09.087313 12733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:09.094676 12733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:09.094789 12733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:09.094874 12733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:09.094980 12733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:09.095060 12733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:09.105533 12733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:56307/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42023
--ranger_config_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:09.106889 12733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:09.108209 12733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:12:09.120966 12733 server_base.cc:1034] running on GCE node
W20260924 03:12:09.120967 12739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:09.121292 12741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:09.121016 12738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:09.122323 12733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:09.123764 12733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:09.125018 12733 hybrid_clock.cc:648] HybridClock initialized: now 1790219529124939 us; error 36 us; skew 500 ppm
I20260924 03:12:09.125360 12733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:09 dist-test-slave-5l98 krb5kdc[12289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:09.128727 12733 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:12:09.130209 12733 webserver.cc:460] Webserver started at http://127.0.97.62:43949/ using document root <none> and password file <none>
I20260924 03:12:09.130844 12733 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:09.130990 12733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:09.131294 12733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:12:09 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 03:12:09.199759 12733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "82d0ed0de9a44703a9db9f28e4c05f52"
format_stamp: "Formatted at 2026-09-24 03:12:09 on dist-test-slave-5l98"
server_key: "0bfb7df2ab99cf81888ddf1ca6678fa6"
server_key_iv: "6c612388979ce40357b23d8bce59a37a"
server_key_version: "kuduclusterkey@0"
I20260924 03:12:09.200925 12733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "82d0ed0de9a44703a9db9f28e4c05f52"
format_stamp: "Formatted at 2026-09-24 03:12:09 on dist-test-slave-5l98"
server_key: "0bfb7df2ab99cf81888ddf1ca6678fa6"
server_key_iv: "6c612388979ce40357b23d8bce59a37a"
server_key_version: "kuduclusterkey@0"
I20260924 03:12:09.207086 12733 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260924 03:12:09.228386 12748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:09.230321 12733 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 03:12:09.230564 12733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "82d0ed0de9a44703a9db9f28e4c05f52"
format_stamp: "Formatted at 2026-09-24 03:12:09 on dist-test-slave-5l98"
server_key: "0bfb7df2ab99cf81888ddf1ca6678fa6"
server_key_iv: "6c612388979ce40357b23d8bce59a37a"
server_key_version: "kuduclusterkey@0"
I20260924 03:12:09.232767 12733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:09.257534 12733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:09.262099 12733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:09.262460 12733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:09.282898 12733 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42023
I20260924 03:12:09.282917 12799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42023 every 8 connection(s)
I20260924 03:12:09.284435 12733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:09.288774 12800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:09.294665 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 12733
I20260924 03:12:09.294919 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:12:09.295575 388 external_mini_cluster.cc:1428] Setting key 21d157d881b3e5aba2a7f5368c4da58c
I20260924 03:12:09.299993 12800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52: Bootstrap starting.
I20260924 03:12:09.303840 12800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:09.305147 12800 log.cc:824] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:09.308559 12800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52: No bootstrap required, opened a new log
Sep 24 03:12:09 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:12:09.319254 12800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d0ed0de9a44703a9db9f28e4c05f52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42023 } }
I20260924 03:12:09.319828 12800 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:09.319952 12800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d0ed0de9a44703a9db9f28e4c05f52, State: Initialized, Role: FOLLOWER
I20260924 03:12:09.320684 12800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d0ed0de9a44703a9db9f28e4c05f52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42023 } }
I20260924 03:12:09.320914 12803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:09.297768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49914 (local address 127.0.97.62:42023)
0924 03:12:09.298608 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:09.298620 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:09.298661 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:12:09.300450 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:09.300474 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:09.300943 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:09.301266 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:09.302989 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:09.304477 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:09.306343 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:09.307107 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:09.307243 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:09.311969 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:09.311980 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:09.312002 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:09.312062 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:09.316191 (+ 4129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:09.316994 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:09.317016 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:09.317027 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:09.317225 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:09.317582 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:09.317587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:09.317591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:09.318337 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:09.318626 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:09.319037 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 03:12:09.319444 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":397,"threads_started":1}
I20260924 03:12:09.320986 12800 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:09.321108 12800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:09.321342 12800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:09.323813 12800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d0ed0de9a44703a9db9f28e4c05f52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42023 } }
I20260924 03:12:09.324308 12800 leader_election.cc:302] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82d0ed0de9a44703a9db9f28e4c05f52; no voters:
I20260924 03:12:09.324954 12800 leader_election.cc:288] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:09.325121 12805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:09.325889 12805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [term 1 LEADER]: Becoming Leader. State: Replica: 82d0ed0de9a44703a9db9f28e4c05f52, State: Running, Role: LEADER
I20260924 03:12:09.326561 12805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d0ed0de9a44703a9db9f28e4c05f52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42023 } }
I20260924 03:12:09.326905 12800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:09.329872 12806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82d0ed0de9a44703a9db9f28e4c05f52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d0ed0de9a44703a9db9f28e4c05f52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42023 } } }
I20260924 03:12:09.330250 12806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:09.329973 12807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82d0ed0de9a44703a9db9f28e4c05f52. Latest consensus state: current_term: 1 leader_uuid: "82d0ed0de9a44703a9db9f28e4c05f52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d0ed0de9a44703a9db9f28e4c05f52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42023 } } }
I20260924 03:12:09.330375 12807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:09.332283 12812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:09.339131 12812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:09.340442 12800 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 03:12:09.358418 12812 catalog_manager.cc:1347] Generated new cluster ID: 1b58fc08d9c54e4ea87597db3da7aa04
I20260924 03:12:09.358587 12812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:09.374367 12812 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:09.376528 12812 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:09.385385 12812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52: Generated new TSK 0
I20260924 03:12:09.386574 12812 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 03:12:10.158265 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42023
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:12:10 dist-test-slave-5l98 krb5kdc[12289](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790219530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 03:12:10.584271 12845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:10.584761 12845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:10.584936 12845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:10.594787 12845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:10.594949 12845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:10.595132 12845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:12:10.607347 12845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:56307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:10.608884 12845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:10.610249 12845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:10.630155 12855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:10.630977 12856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:10.639442 12858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:10.640918 12845 server_base.cc:1034] running on GCE node
I20260924 03:12:10.641682 12845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:10.642835 12845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:10.644124 12845 hybrid_clock.cc:648] HybridClock initialized: now 1790219530644050 us; error 60 us; skew 500 ppm
I20260924 03:12:10.644593 12845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:10 dist-test-slave-5l98 krb5kdc[12289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:10.648168 12845 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:12:10.654100 12845 webserver.cc:460] Webserver started at http://127.0.97.1:36543/ using document root <none> and password file <none>
I20260924 03:12:10.654992 12845 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:10.655148 12845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:10.655475 12845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:12:10 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 03:12:10.674237 12845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "caec79b7d075429f9118bc256ccaa3d0"
format_stamp: "Formatted at 2026-09-24 03:12:10 on dist-test-slave-5l98"
server_key: "a47775e7af619fd6d537c5a653278fac"
server_key_iv: "34d2f64b192f2c7d5a23140fa622b8d2"
server_key_version: "kuduclusterkey@0"
I20260924 03:12:10.675534 12845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "caec79b7d075429f9118bc256ccaa3d0"
format_stamp: "Formatted at 2026-09-24 03:12:10 on dist-test-slave-5l98"
server_key: "a47775e7af619fd6d537c5a653278fac"
server_key_iv: "34d2f64b192f2c7d5a23140fa622b8d2"
server_key_version: "kuduclusterkey@0"
I20260924 03:12:10.701150 12845 fs_manager.cc:691] Time spent creating directory manager: real 0.025s user 0.013s sys 0.008s
I20260924 03:12:10.729902 12865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:10.733793 12845 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.009s sys 0.003s
I20260924 03:12:10.734148 12845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "caec79b7d075429f9118bc256ccaa3d0"
format_stamp: "Formatted at 2026-09-24 03:12:10 on dist-test-slave-5l98"
server_key: "a47775e7af619fd6d537c5a653278fac"
server_key_iv: "34d2f64b192f2c7d5a23140fa622b8d2"
server_key_version: "kuduclusterkey@0"
I20260924 03:12:10.734525 12845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:10.752455 12845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:10.758061 12845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:10.758476 12845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:10.759754 12845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:10.761703 12845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:10.761835 12845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:10.761989 12845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:10.762079 12845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:10.843307 12845 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43147
I20260924 03:12:10.843324 12977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43147 every 8 connection(s)
I20260924 03:12:10.844897 12845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:12:10.852028 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 12845
I20260924 03:12:10.852279 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:12:10.853338 388 external_mini_cluster.cc:1428] Setting key 8e5d5fcd854bb5fcff1def8c790da586
Sep 24 03:12:10 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 03:12:10.898227 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:10.851649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42023 (local address 127.0.97.1:52269)
0924 03:12:10.862420 (+ 10771us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:10.862509 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:10.868297 (+ 5788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:10.870556 (+ 2259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:10.870569 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:10.871259 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:10.872150 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:10.872186 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:10.873746 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:10.873752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:10.875858 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:10.875878 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:10.877024 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:10.877033 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:10.877325 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:10.878199 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:10.878251 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:10.882556 (+ 4305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:10.892277 (+ 9721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:10.892293 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:10.892318 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:10.893197 (+ 879us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:10.893818 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:10.893823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:10.893828 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:10.894135 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:10.895095 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:10.895107 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:10.895665 (+ 558us) client_negotiation.cc:769] Sending connection context
0924 03:12:10.895992 (+ 327us) client_negotiation.cc:240] Negotiation successful
0924 03:12:10.896660 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10268,"thread_start_us":13957,"threads_started":1}
I20260924 03:12:10.899750 12980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:10.850751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52269 (local address 127.0.97.62:42023)
0924 03:12:10.869809 (+ 19058us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:10.869816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:10.869838 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:12:10.869908 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:10.869914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:10.870002 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:10.870142 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:10.872387 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:10.873554 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:10.876081 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:10.876754 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:10.877787 (+ 1033us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:10.882878 (+ 5091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:10.882888 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:10.882896 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:10.882960 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:10.891913 (+ 8953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:10.893452 (+ 1539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:10.893461 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:10.893468 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:10.893662 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:10.894316 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:10.894324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:10.894431 (+ 107us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:10.894917 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:10.899153 (+ 4236us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:10.899285 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 03:12:10.899489 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18911,"thread_start_us":19507,"threads_started":1}
I20260924 03:12:10.901708 12978 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 03:12:10.902113 12978 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:10.903060 12978 heartbeater.cc:511] Master 127.0.97.62:42023 requested a full tablet report, sending...
I20260924 03:12:10.909381 12765 ts_manager.cc:194] Registered new tserver with Master: caec79b7d075429f9118bc256ccaa3d0 (127.0.97.1:43147)
I20260924 03:12:10.911718 12765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 03:12:10.938354 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42023
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:12:10 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 24 03:12:11 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260924 03:12:11.432294 12984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:11.432823 12984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:11.433022 12984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:11.444557 12984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:11.444785 12984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:11.445034 12984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:12:11.460173 12984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:56307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:11.461800 12984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:11.463413 12984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:11.505709 12991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:11.506541 12992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:11.513453 12994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:11.514377 12984 server_base.cc:1034] running on GCE node
I20260924 03:12:11.515396 12984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:11.523458 12984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:11.526944 12984 hybrid_clock.cc:648] HybridClock initialized: now 1790219531526808 us; error 153 us; skew 500 ppm
I20260924 03:12:11.527453 12984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:11 dist-test-slave-5l98 krb5kdc[12289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:11.531738 12984 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:12:11.559371 12984 webserver.cc:460] Webserver started at http://127.0.97.2:36355/ using document root <none> and password file <none>
I20260924 03:12:11.560333 12984 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:11.560541 12984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:11.560904 12984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:12:11 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 03:12:11.579170 12984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "635d5ccfe07c477587d79700b550e7f7"
format_stamp: "Formatted at 2026-09-24 03:12:11 on dist-test-slave-5l98"
server_key: "d850d965e86c7729c6b1690a6f5459b5"
server_key_iv: "09ee0f1af8293c8de3c8342c69f5a4d1"
server_key_version: "kuduclusterkey@0"
I20260924 03:12:11.580292 12984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "635d5ccfe07c477587d79700b550e7f7"
format_stamp: "Formatted at 2026-09-24 03:12:11 on dist-test-slave-5l98"
server_key: "d850d965e86c7729c6b1690a6f5459b5"
server_key_iv: "09ee0f1af8293c8de3c8342c69f5a4d1"
server_key_version: "kuduclusterkey@0"
I20260924 03:12:11.603116 12984 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.012s sys 0.008s
I20260924 03:12:11.636706 13008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:11.638528 12984 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260924 03:12:11.638731 12984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "635d5ccfe07c477587d79700b550e7f7"
format_stamp: "Formatted at 2026-09-24 03:12:11 on dist-test-slave-5l98"
server_key: "d850d965e86c7729c6b1690a6f5459b5"
server_key_iv: "09ee0f1af8293c8de3c8342c69f5a4d1"
server_key_version: "kuduclusterkey@0"
I20260924 03:12:11.638973 12984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:11.651690 12984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:11.656112 12984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:11.656433 12984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:11.657521 12984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:11.659149 12984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:11.659252 12984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:11.659386 12984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:11.659466 12984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:11.709158 12984 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40209
I20260924 03:12:11.709192 13120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40209 every 8 connection(s)
I20260924 03:12:11.710954 12984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:12:11.716063 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 12984
I20260924 03:12:11.716277 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:12:11.717000 388 external_mini_cluster.cc:1428] Setting key f27af34fc2465d03ec9b4320457e739f
Sep 24 03:12:11 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 03:12:11.769991 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.715165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42023 (local address 127.0.97.2:47687)
0924 03:12:11.741389 (+ 26224us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:11.741457 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:11.744036 (+ 2579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:11.745562 (+ 1526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:11.745575 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:11.746268 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:11.747139 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:11.747163 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.752316 (+ 5153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.752322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:11.754509 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:11.754527 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.755780 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.755787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:11.756102 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.757304 (+ 1202us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:11.757334 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:11.760365 (+ 3031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:11.764570 (+ 4205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:11.764581 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:11.764604 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:11.765354 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:11.765867 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:11.765872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:11.765875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:11.766261 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:11.767199 (+ 938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:11.767208 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:11.767653 (+ 445us) client_negotiation.cc:769] Sending connection context
0924 03:12:11.767958 (+ 305us) client_negotiation.cc:240] Negotiation successful
0924 03:12:11.768561 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25865,"thread_start_us":25595,"threads_started":1}
I20260924 03:12:11.771395 13123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.714325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47687 (local address 127.0.97.62:42023)
0924 03:12:11.744946 (+ 30621us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:11.744952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:11.744973 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:12:11.745037 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:11.745042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:11.745132 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:11.750806 (+ 5674us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:11.751005 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.752102 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.754733 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.755340 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.755462 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.760707 (+ 5245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:11.760715 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:11.760720 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:11.760764 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:11.764336 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:11.765542 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:11.765548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:11.765552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:11.765728 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:11.766441 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:11.766446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:11.766449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:11.766926 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:11.767124 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:11.770760 (+ 3636us) server_negotiation.cc:299] Negotiation successful
0924 03:12:11.771030 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30473,"thread_start_us":33864,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 03:12:11.772897 13121 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42023
I20260924 03:12:11.773240 13121 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:11.774101 13121 heartbeater.cc:511] Master 127.0.97.62:42023 requested a full tablet report, sending...
I20260924 03:12:11.775915 12765 ts_manager.cc:194] Registered new tserver with Master: 635d5ccfe07c477587d79700b550e7f7 (127.0.97.2:40209)
I20260924 03:12:11.777122 12765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 03:12:11.810256 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42023
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:12:11.916160 12978 heartbeater.cc:503] Master 127.0.97.62:42023 was elected leader, sending a full tablet report...
W20260924 03:12:12.180299 13127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:12.180651 13127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:12.180845 13127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:12.188205 13127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:12.188313 13127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:12.188480 13127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:12:12.202991 13127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:56307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:12.204355 13127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:12.205695 13127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:12.219425 13132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:12.219434 13135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:12.219601 13133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:12.220396 13127 server_base.cc:1034] running on GCE node
I20260924 03:12:12.221135 13127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:12.222195 13127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:12.223423 13127 hybrid_clock.cc:648] HybridClock initialized: now 1790219532223433 us; error 105 us; skew 500 ppm
I20260924 03:12:12.223750 13127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:12 dist-test-slave-5l98 krb5kdc[12289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:12.227205 13127 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:12:12.228843 13127 webserver.cc:460] Webserver started at http://127.0.97.3:46823/ using document root <none> and password file <none>
I20260924 03:12:12.229550 13127 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:12.229693 13127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:12.230002 13127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:12:12 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 03:12:12.246142 13127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "b0755c3a209642c78efe7641bcaf6158"
format_stamp: "Formatted at 2026-09-24 03:12:12 on dist-test-slave-5l98"
server_key: "0e7ba3febde89c42da38edba02f15926"
server_key_iv: "cc039230c4304f840d713b66e7d45c12"
server_key_version: "kuduclusterkey@0"
I20260924 03:12:12.247016 13127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b0755c3a209642c78efe7641bcaf6158"
format_stamp: "Formatted at 2026-09-24 03:12:12 on dist-test-slave-5l98"
server_key: "0e7ba3febde89c42da38edba02f15926"
server_key_iv: "cc039230c4304f840d713b66e7d45c12"
server_key_version: "kuduclusterkey@0"
I20260924 03:12:12.252545 13127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 03:12:12.271270 13142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:12.273110 13127 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:12:12.273321 13127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "b0755c3a209642c78efe7641bcaf6158"
format_stamp: "Formatted at 2026-09-24 03:12:12 on dist-test-slave-5l98"
server_key: "0e7ba3febde89c42da38edba02f15926"
server_key_iv: "cc039230c4304f840d713b66e7d45c12"
server_key_version: "kuduclusterkey@0"
I20260924 03:12:12.273552 13127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:12.300704 13127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:12.304939 13127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:12.305261 13127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:12.306221 13127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:12.307606 13127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:12.307678 13127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:12.307803 13127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:12.307852 13127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:12.342697 13127 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45531
I20260924 03:12:12.342780 13254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45531 every 8 connection(s)
I20260924 03:12:12.344139 13127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:12:12.350171 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 13127
I20260924 03:12:12.350394 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:12:12.350881 388 external_mini_cluster.cc:1428] Setting key 245189d497c2b668f012c79028db730c
Sep 24 03:12:12 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:12:12.369899 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.347047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55935 (local address 127.0.97.62:42023)
0924 03:12:12.347541 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:12.347546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:12.348604 (+ 1058us) server_negotiation.cc:407] Connection header received
0924 03:12:12.350428 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:12.350434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:12.350537 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:12.350660 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:12.352210 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.353047 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.354571 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.354994 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.355126 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:12.358573 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:12.358578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:12.358582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:12.358614 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:12.366497 (+ 7883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:12.367358 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:12.367364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:12.367367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:12.367517 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:12.368015 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:12.368019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:12.368021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:12.368325 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:12.368453 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:12.369417 (+ 964us) server_negotiation.cc:299] Negotiation successful
0924 03:12:12.369625 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":270,"threads_started":1}
I20260924 03:12:12.371012 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.347525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42023 (local address 127.0.97.3:55935)
0924 03:12:12.348370 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:12.348408 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:12.350128 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:12.350892 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:12.350900 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:12.351371 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:12.352032 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:12.352058 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.353203 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.353208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:12.354425 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:12.354436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.355218 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.355225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:12.355460 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:12.356156 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:12.356178 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:12.358342 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:12.366688 (+ 8346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:12.366698 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:12.366719 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:12.367222 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:12.367650 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:12.367654 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:12.367656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:12.367875 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:12.368653 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:12.368660 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:12.369030 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 03:12:12.369347 (+ 317us) client_negotiation.cc:240] Negotiation successful
0924 03:12:12.369763 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":363,"threads_started":1}
I20260924 03:12:12.372570 13255 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42023
I20260924 03:12:12.372900 13255 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:12.373667 13255 heartbeater.cc:511] Master 127.0.97.62:42023 requested a full tablet report, sending...
I20260924 03:12:12.375216 12765 ts_manager.cc:194] Registered new tserver with Master: b0755c3a209642c78efe7641bcaf6158 (127.0.97.3:45531)
I20260924 03:12:12.376151 12765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55935
I20260924 03:12:12.379781 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:12:12.381619 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 12845
Sep 24 03:12:12 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:12:12.399032 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.382663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56456 (local address 127.0.97.1:43147)
0924 03:12:12.383516 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:12.383524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:12.383548 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:12:12.383603 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:12.383610 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:12.383805 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:12.383984 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:12.385582 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.386597 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.388124 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.388736 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.388876 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:12.393126 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:12.393136 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:12.393149 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:12.393181 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:12.396148 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:12.396929 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:12.396939 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:12.396949 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:12.397110 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:12.397492 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:12.397501 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:12.397503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:12.397823 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:12.398150 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:12.398433 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 03:12:12.398691 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":385,"threads_started":1}
I20260924 03:12:12.400389 12973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:12.486913 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 12845
I20260924 03:12:12.502732 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 12984
Sep 24 03:12:12 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 03:12:12.519178 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.503619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45734 (local address 127.0.97.2:40209)
0924 03:12:12.504222 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:12.504230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:12.504255 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:12:12.504314 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:12.504321 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:12.504588 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:12.504728 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:12.505965 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.506969 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.508473 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.509113 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.509216 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:12.513538 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:12.513550 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:12.513563 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:12.513606 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:12.516270 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:12.517068 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:12.517074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:12.517076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:12.517211 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:12.517649 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:12.517653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:12.517655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:12.518047 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:12.518337 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:12.518507 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 03:12:12.518729 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":343,"threads_started":1}
I20260924 03:12:12.520712 13116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:12.604962 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 12984
I20260924 03:12:12.624423 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 13127
Sep 24 03:12:12 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 03:12:12.642596 13264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.625527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54870 (local address 127.0.97.3:45531)
0924 03:12:12.626078 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:12.626086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:12.626115 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:12:12.626175 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:12.626181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:12.626398 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:12.626576 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:12.627955 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.629125 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.630395 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.630959 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.631104 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:12.636217 (+ 5113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:12.636229 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:12.636244 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:12.636279 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:12.639551 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:12.640331 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:12.640339 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:12.640342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:12.640469 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:12.640959 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:12.640965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:12.640970 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:12.641314 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:12.641647 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:12.641949 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 03:12:12.642210 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":244,"threads_started":1}
I20260924 03:12:12.644034 13250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:12.727931 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 13127
I20260924 03:12:12.743737 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 12733
I20260924 03:12:12.745146 12795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:12.853914 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 12733
I20260924 03:12:12.876307 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42023
--webserver_interface=127.0.97.62
--webserver_port=43949
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42023
--ranger_config_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56307/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:56307/kms with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:13.211156 13267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:13.211465 13267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:13.211629 13267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:13.218855 13267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:13.218966 13267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:13.219058 13267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:13.219163 13267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:13.219235 13267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:13.229643 13267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:56307/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42023
--ranger_config_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:13.230957 13267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:13.232304 13267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:13.246356 13272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:13.246438 13275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:13.246408 13273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:13.247787 13267 server_base.cc:1034] running on GCE node
I20260924 03:12:13.248703 13267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:13.250336 13267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:13.251556 13267 hybrid_clock.cc:648] HybridClock initialized: now 1790219533251501 us; error 49 us; skew 500 ppm
I20260924 03:12:13.251936 13267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:13 dist-test-slave-5l98 krb5kdc[12289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:13.255256 13267 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:12:13.256855 13267 webserver.cc:460] Webserver started at http://127.0.97.62:43949/ using document root <none> and password file <none>
I20260924 03:12:13.257558 13267 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:13.257671 13267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 03:12:13 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 03:12:13.308140 13267 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:12:13.311465 13283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:13.313251 13267 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260924 03:12:13.313465 13267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "82d0ed0de9a44703a9db9f28e4c05f52"
format_stamp: "Formatted at 2026-09-24 03:12:09 on dist-test-slave-5l98"
server_key: "0bfb7df2ab99cf81888ddf1ca6678fa6"
server_key_iv: "6c612388979ce40357b23d8bce59a37a"
server_key_version: "kuduclusterkey@0"
I20260924 03:12:13.314491 13267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:13.329057 13267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:13.333249 13267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:13.333552 13267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:13.353121 13267 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42023
I20260924 03:12:13.353144 13334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42023 every 8 connection(s)
I20260924 03:12:13.354646 13267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:13.361143 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 13267
I20260924 03:12:13.362233 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43147
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42023
--webserver_port=36543
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:56307/kms with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:12:13.363260 13335 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:12:13.369443 13335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52: Bootstrap starting.
I20260924 03:12:13.389559 13335 log.cc:824] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:13.398490 13335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:12:13.399057 13335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52: Bootstrap complete.
I20260924 03:12:13.408341 13335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d0ed0de9a44703a9db9f28e4c05f52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42023 } }
I20260924 03:12:13.409372 13335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d0ed0de9a44703a9db9f28e4c05f52, State: Initialized, Role: FOLLOWER
I20260924 03:12:13.410090 13335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d0ed0de9a44703a9db9f28e4c05f52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42023 } }
I20260924 03:12:13.410332 13335 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:13.410450 13335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:13.410636 13335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:12:13.414736 13335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d0ed0de9a44703a9db9f28e4c05f52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42023 } }
I20260924 03:12:13.415284 13335 leader_election.cc:302] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82d0ed0de9a44703a9db9f28e4c05f52; no voters:
I20260924 03:12:13.415923 13335 leader_election.cc:288] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 03:12:13.416083 13339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:12:13.417186 13339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [term 2 LEADER]: Becoming Leader. State: Replica: 82d0ed0de9a44703a9db9f28e4c05f52, State: Running, Role: LEADER
I20260924 03:12:13.417830 13339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d0ed0de9a44703a9db9f28e4c05f52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42023 } }
I20260924 03:12:13.418444 13335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:13.420429 13341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82d0ed0de9a44703a9db9f28e4c05f52. Latest consensus state: current_term: 2 leader_uuid: "82d0ed0de9a44703a9db9f28e4c05f52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d0ed0de9a44703a9db9f28e4c05f52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42023 } } }
I20260924 03:12:13.420897 13341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:13.421509 13340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "82d0ed0de9a44703a9db9f28e4c05f52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d0ed0de9a44703a9db9f28e4c05f52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42023 } } }
I20260924 03:12:13.421828 13340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:13.429072 13345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:13.434801 13345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:13.436250 13345 catalog_manager.cc:1259] Loaded cluster ID: 1b58fc08d9c54e4ea87597db3da7aa04
I20260924 03:12:13.436410 13345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:13.440613 13345 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:13.442687 13345 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52: Loaded TSK: 0
I20260924 03:12:13.444408 13345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:13.446882 13335 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260924 03:12:13.824690 13337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:13.825201 13337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:13.825405 13337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:13.837282 13337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:13.837507 13337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:13.837739 13337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:12:13.853878 13337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:56307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36543
--tserver_master_addrs=127.0.97.62:42023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:13.855954 13337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:13.857882 13337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:13.884948 13373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:13.885988 13374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:13.890327 13376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:13.891296 13337 server_base.cc:1034] running on GCE node
I20260924 03:12:13.892436 13337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:13.894183 13337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:13.895501 13337 hybrid_clock.cc:648] HybridClock initialized: now 1790219533895420 us; error 69 us; skew 500 ppm
I20260924 03:12:13.895931 13337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:13 dist-test-slave-5l98 krb5kdc[12289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:13.900599 13337 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:12:13.902853 13337 webserver.cc:460] Webserver started at http://127.0.97.1:36543/ using document root <none> and password file <none>
I20260924 03:12:13.903930 13337 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:13.904129 13337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 03:12:13 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 03:12:13.945654 13337 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.006s sys 0.001s
I20260924 03:12:13.961331 13384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:13.963587 13337 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.003s sys 0.012s
I20260924 03:12:13.966267 13337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "caec79b7d075429f9118bc256ccaa3d0"
format_stamp: "Formatted at 2026-09-24 03:12:10 on dist-test-slave-5l98"
server_key: "a47775e7af619fd6d537c5a653278fac"
server_key_iv: "34d2f64b192f2c7d5a23140fa622b8d2"
server_key_version: "kuduclusterkey@0"
I20260924 03:12:13.967979 13337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:14.017897 13337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:14.023583 13337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:14.023977 13337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:14.031719 13337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:14.040082 13337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:14.040254 13337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:14.040400 13337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:14.040540 13337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:14.119444 13497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43147 every 8 connection(s)
I20260924 03:12:14.119445 13337 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43147
I20260924 03:12:14.121335 13337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:12:14.128108 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 13337
I20260924 03:12:14.129016 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40209
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42023
--webserver_port=36355
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:56307/kms with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:12:14 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:12:14.166710 13501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.129541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35437 (local address 127.0.97.62:42023)
0924 03:12:14.136759 (+ 7218us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:14.136783 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:14.144365 (+ 7582us) server_negotiation.cc:407] Connection header received
0924 03:12:14.144835 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:14.144849 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:14.145266 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:14.145659 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:14.147154 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.148396 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.150118 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.150785 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.151012 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.154753 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:14.154761 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:14.154778 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:14.154823 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:14.159512 (+ 4689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:14.160739 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:14.160750 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:14.160760 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:14.160988 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:14.161593 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:14.161598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:14.161602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:14.162284 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:14.162510 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:14.164835 (+ 2325us) server_negotiation.cc:299] Negotiation successful
0924 03:12:14.165300 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6831,"thread_start_us":9282,"threads_started":1}
I20260924 03:12:14.167846 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.128183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42023 (local address 127.0.97.1:35437)
0924 03:12:14.141261 (+ 13078us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:14.141329 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:14.143920 (+ 2591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:14.145817 (+ 1897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:14.145830 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:14.146380 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:14.146929 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.146949 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.148603 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.148611 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.149948 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.149962 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.151037 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.151046 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.151326 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.152059 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:14.152084 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:14.154421 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:14.159762 (+ 5341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:14.159774 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:14.159791 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:14.160545 (+ 754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:14.161147 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:14.161152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:14.161156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:14.161441 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:14.162449 (+ 1008us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:14.162458 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:14.162784 (+ 326us) client_negotiation.cc:769] Sending connection context
0924 03:12:14.163121 (+ 337us) client_negotiation.cc:240] Negotiation successful
0924 03:12:14.163721 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12743,"thread_start_us":12474,"threads_started":1}
I20260924 03:12:14.172684 13498 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42023
I20260924 03:12:14.173053 13498 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:14.173995 13498 heartbeater.cc:511] Master 127.0.97.62:42023 requested a full tablet report, sending...
W20260924 03:12:14.177682 13498 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42023 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 03:12:14.177960 13498 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:14.178289 13498 heartbeater.cc:511] Master 127.0.97.62:42023 requested a full tablet report, sending...
I20260924 03:12:14.179764 13498 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:14.180136 13498 heartbeater.cc:511] Master 127.0.97.62:42023 requested a full tablet report, sending...
W20260924 03:12:14.181432 13498 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:12:14.570475 13502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:14.570935 13502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:14.571133 13502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:14.580339 13502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:14.580463 13502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:14.580668 13502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:12:14.596088 13502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:56307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36355
--tserver_master_addrs=127.0.97.62:42023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:14.598237 13502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:14.600039 13502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:14.617297 13516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:14.617623 13517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:14.619065 13519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:14.619781 13502 server_base.cc:1034] running on GCE node
I20260924 03:12:14.620638 13502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:14.628321 13502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:14.629729 13502 hybrid_clock.cc:648] HybridClock initialized: now 1790219534629673 us; error 108 us; skew 500 ppm
I20260924 03:12:14.630154 13502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:14 dist-test-slave-5l98 krb5kdc[12289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:14.634668 13502 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:12:14.665251 13502 webserver.cc:460] Webserver started at http://127.0.97.2:36355/ using document root <none> and password file <none>
I20260924 03:12:14.666545 13502 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:14.666749 13502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 03:12:14 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 03:12:14.706736 13502 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.010s sys 0.007s
I20260924 03:12:14.721999 13529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:14.724110 13502 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.005s sys 0.008s
I20260924 03:12:14.724387 13502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "635d5ccfe07c477587d79700b550e7f7"
format_stamp: "Formatted at 2026-09-24 03:12:11 on dist-test-slave-5l98"
server_key: "d850d965e86c7729c6b1690a6f5459b5"
server_key_iv: "09ee0f1af8293c8de3c8342c69f5a4d1"
server_key_version: "kuduclusterkey@0"
I20260924 03:12:14.726015 13502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:14.767653 13502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Sep 24 03:12:14 dist-test-slave-5l98 krb5kdc[12289](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790219534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:14.772760 13502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:14.773097 13502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:14.774276 13502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:14.778234 13502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:14.778363 13502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:14.778470 13502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:14.778539 13502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:14.852907 13502 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40209
I20260924 03:12:14.852958 13641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40209 every 8 connection(s)
I20260924 03:12:14.854833 13502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:12:14.864368 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 13502
I20260924 03:12:14.865208 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:56307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:45531
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42023
--webserver_port=46823
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:56307/kms with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:12:14 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:12:14.892668 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.858029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34899 (local address 127.0.97.62:42023)
0924 03:12:14.858555 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:14.858562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:14.866346 (+ 7784us) server_negotiation.cc:407] Connection header received
0924 03:12:14.869420 (+ 3074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:14.869427 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:14.869526 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:14.869659 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:14.872138 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.873370 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.876197 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.876924 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.877043 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.884228 (+ 7185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:14.884236 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:14.884240 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:14.884291 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:14.888273 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:14.889406 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:14.889412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:14.889425 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:14.889592 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:14.890196 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:14.890200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:14.890203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:14.890587 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:14.890714 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:14.891946 (+ 1232us) server_negotiation.cc:299] Negotiation successful
0924 03:12:14.892208 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":258,"threads_started":1}
I20260924 03:12:14.894233 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.858651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42023 (local address 127.0.97.2:34899)
0924 03:12:14.866120 (+ 7469us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:14.866187 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:14.868979 (+ 2792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:14.870331 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:14.870344 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:14.871023 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:14.871890 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.871929 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.873591 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.873596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.876007 (+ 2411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.876033 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.877433 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.877441 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.877747 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.878591 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:14.878626 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:14.883808 (+ 5182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:14.888528 (+ 4720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:14.888539 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:14.888561 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:14.889215 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:14.889751 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:14.889756 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:14.889759 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:14.890054 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:14.890823 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:14.890833 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:14.891402 (+ 569us) client_negotiation.cc:769] Sending connection context
0924 03:12:14.891778 (+ 376us) client_negotiation.cc:240] Negotiation successful
0924 03:12:14.892538 (+ 760us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6333,"thread_start_us":355,"threads_started":1}
I20260924 03:12:14.895928 13643 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42023
I20260924 03:12:14.896267 13643 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:14.897078 13643 heartbeater.cc:511] Master 127.0.97.62:42023 requested a full tablet report, sending...
I20260924 03:12:14.899050 13300 ts_manager.cc:194] Registered new tserver with Master: 635d5ccfe07c477587d79700b550e7f7 (127.0.97.2:40209)
I20260924 03:12:14.901425 13300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34899
I20260924 03:12:15.181571 13498 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:15.182045 13498 heartbeater.cc:511] Master 127.0.97.62:42023 requested a full tablet report, sending...
I20260924 03:12:15.183323 13300 ts_manager.cc:194] Registered new tserver with Master: caec79b7d075429f9118bc256ccaa3d0 (127.0.97.1:43147)
I20260924 03:12:15.184583 13300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35437
Sep 24 03:12:15 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260924 03:12:15.319787 13647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:15.320199 13647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:15.320355 13647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:15.335083 13647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:15.335273 13647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:15.335448 13647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:12:15.346832 13647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:56307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:45531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=46823
--tserver_master_addrs=127.0.97.62:42023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:15.348313 13647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:15.349824 13647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:15.384354 13652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:15.384599 13653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:15.389978 13647 server_base.cc:1034] running on GCE node
W20260924 03:12:15.391209 13655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:15.392069 13647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:15.393716 13647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:15.395013 13647 hybrid_clock.cc:648] HybridClock initialized: now 1790219535394953 us; error 48 us; skew 500 ppm
I20260924 03:12:15.395597 13647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:15 dist-test-slave-5l98 krb5kdc[12289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:15.400139 13647 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:12:15.406221 13647 webserver.cc:460] Webserver started at http://127.0.97.3:46823/ using document root <none> and password file <none>
I20260924 03:12:15.407256 13647 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:15.407433 13647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 03:12:15 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 03:12:15.454957 13647 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.011s sys 0.007s
I20260924 03:12:15.459143 13663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:15.461098 13647 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 03:12:15.461288 13647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "b0755c3a209642c78efe7641bcaf6158"
format_stamp: "Formatted at 2026-09-24 03:12:12 on dist-test-slave-5l98"
server_key: "0e7ba3febde89c42da38edba02f15926"
server_key_iv: "cc039230c4304f840d713b66e7d45c12"
server_key_version: "kuduclusterkey@0"
I20260924 03:12:15.462724 13647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:15.484854 13647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:15.490778 13647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:15.491120 13647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:15.492448 13647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:15.494369 13647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:15.494488 13647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:15.494614 13647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:15.494696 13647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:12:15 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 03:12:15.694059 13647 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45531
I20260924 03:12:15.694147 13776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45531 every 8 connection(s)
I20260924 03:12:15.696085 13647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:12:15.702809 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 13647
Sep 24 03:12:15 dist-test-slave-5l98 krb5kdc[12289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:12:15.738173 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:15.704928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49930 (local address 127.0.97.62:42023)
0924 03:12:15.718319 (+ 13391us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:15.718326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:15.719112 (+ 786us) server_negotiation.cc:407] Connection header received
0924 03:12:15.719313 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:15.719319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:15.719403 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:15.719521 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:15.721728 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.722936 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.725026 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.725670 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.725764 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:15.729504 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:15.729523 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:15.729527 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:15.729585 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:15.733311 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:15.734223 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:15.734228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:15.734231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:15.734386 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:15.734845 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:15.734852 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:15.734855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:15.735454 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:15.735729 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:15.737565 (+ 1836us) server_negotiation.cc:299] Negotiation successful
0924 03:12:15.737794 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13258,"thread_start_us":12644,"threads_started":1}
I20260924 03:12:15.748873 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:15.700786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42023 (local address 127.0.97.3:44955)
0924 03:12:15.709230 (+ 8444us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:15.709326 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:15.717707 (+ 8381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:15.725682 (+ 7975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:15.725697 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:15.726432 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:15.727387 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:15.727423 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.730722 (+ 3299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.730728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:15.733038 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:15.733058 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.734256 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.734265 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:15.734600 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:15.735340 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:15.735370 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:15.739568 (+ 4198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:15.744154 (+ 4586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:15.744166 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:15.744188 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:15.744860 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:15.745314 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:15.745319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:15.745323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:15.745601 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:15.746321 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:15.746330 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:15.746765 (+ 435us) client_negotiation.cc:769] Sending connection context
0924 03:12:15.747089 (+ 324us) client_negotiation.cc:240] Negotiation successful
0924 03:12:15.747663 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8037,"thread_start_us":7756,"threads_started":1}
I20260924 03:12:15.749811 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:15.701601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44955 (local address 127.0.97.62:42023)
0924 03:12:15.713510 (+ 11909us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:15.713516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:15.713541 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:12:15.725051 (+ 11510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:15.725055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:15.725149 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:15.727602 (+ 2453us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:15.727862 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.729225 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.733259 (+ 4034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.733938 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.735003 (+ 1065us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:15.740360 (+ 5357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:15.740367 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:15.740372 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:15.740429 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:15.743994 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:15.745022 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:15.745028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:15.745032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:15.745176 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:15.745744 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:15.745749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:15.745752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:15.746113 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:15.746241 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:15.749398 (+ 3157us) server_negotiation.cc:299] Negotiation successful
0924 03:12:15.749565 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11787,"thread_start_us":11621,"threads_started":1}
I20260924 03:12:15.751758 13777 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42023
I20260924 03:12:15.752104 13777 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:15.753001 13777 heartbeater.cc:511] Master 127.0.97.62:42023 requested a full tablet report, sending...
I20260924 03:12:15.755295 13299 ts_manager.cc:194] Registered new tserver with Master: b0755c3a209642c78efe7641bcaf6158 (127.0.97.3:45531)
I20260924 03:12:15.756347 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:12:15.756726 13299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44955
I20260924 03:12:15.790177 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:15.769839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49942 (local address 127.0.97.62:42023)
0924 03:12:15.770046 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:15.770052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:15.774981 (+ 4929us) server_negotiation.cc:407] Connection header received
0924 03:12:15.775300 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:15.775306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:15.775409 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:15.775489 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:15.777663 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.778803 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.780959 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.781429 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.781514 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:15.784613 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:15.784618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:15.784622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:15.784658 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:15.787324 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:15.788066 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:15.788071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:15.788075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:15.788207 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:15.788647 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:15.788651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:15.788653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:15.788944 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:15.789203 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:15.789582 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 03:12:15.789832 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:12:15.795385 13299 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:12:15.803061 13299 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:12:15.834446 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:15.822741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45531 (local address 127.0.0.1:54880)
0924 03:12:15.824301 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:15.824337 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:15.824545 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:15.826051 (+ 1506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:15.826058 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:15.826098 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:15.826542 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:15.826557 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.829635 (+ 3078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.829644 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:15.831424 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:15.831445 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.833404 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.833410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:15.833644 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:15.833716 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 03:12:15.833821 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 03:12:15.834024 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1335,"thread_start_us":8941,"threads_started":1}
I20260924 03:12:15.836198 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:15.823916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54880 (local address 127.0.97.3:45531)
0924 03:12:15.825457 (+ 1541us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:15.825463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:15.825489 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:12:15.825632 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:15.825638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:15.825815 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:15.826011 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:15.826749 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.829487 (+ 2738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.831669 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.833193 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.834995 (+ 1802us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:15.835235 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:15.835312 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 03:12:15.835475 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1390,"thread_start_us":1242,"threads_started":1}
I20260924 03:12:15.840456 13712 tablet_service.cc:1467] Processing CreateTablet for tablet c061130f43d444248e147a2d54aeb5f1 (DEFAULT_TABLE table=test-table [id=cdab5c2e6a444b7887de4d0b98812ea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:15.842379 13712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c061130f43d444248e147a2d54aeb5f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:15.844269 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:15.835601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40209 (local address 127.0.0.1:45748)
0924 03:12:15.835772 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:15.835790 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:15.835884 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:15.837861 (+ 1977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:15.837866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:15.837894 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:15.838102 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:15.838115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.839906 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.839911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:15.841728 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:15.841746 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.843578 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.843585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:15.843774 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:15.843804 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 03:12:15.843871 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 03:12:15.844016 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":15}
I20260924 03:12:15.844509 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:15.823414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56464 (local address 127.0.97.1:43147)
0924 03:12:15.826748 (+ 3334us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:15.826756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:15.833084 (+ 6328us) server_negotiation.cc:407] Connection header received
0924 03:12:15.833300 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:15.833308 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:15.833600 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:15.833791 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:15.834666 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.836189 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.839932 (+ 3743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.841427 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.841545 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:15.841769 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:15.843977 (+ 2208us) server_negotiation.cc:299] Negotiation successful
0924 03:12:15.844164 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3165,"thread_start_us":2978,"threads_started":1}
I20260924 03:12:15.844803 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:15.823124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43147 (local address 127.0.0.1:56464)
0924 03:12:15.831781 (+ 8657us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:15.831805 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:15.831929 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:15.834040 (+ 2111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:15.834045 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:15.834069 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:15.834438 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:15.834451 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.836349 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.836354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:15.839749 (+ 3395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:15.839761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.841909 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.841915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:15.842129 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:15.842165 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 03:12:15.842256 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 03:12:15.842392 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8514,"thread_start_us":809,"threads_started":1}
I20260924 03:12:15.845327 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:15.832395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.0.97.2:40209)
0924 03:12:15.837274 (+ 4879us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:15.837282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:15.837309 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:12:15.837399 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:15.837407 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:15.837656 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:15.837993 (+ 337us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:15.838224 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.839758 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.841935 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.843396 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.844533 (+ 1137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:15.844750 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:15.844816 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 03:12:15.844999 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4700,"threads_started":1}
I20260924 03:12:15.848807 13433 tablet_service.cc:1467] Processing CreateTablet for tablet c061130f43d444248e147a2d54aeb5f1 (DEFAULT_TABLE table=test-table [id=cdab5c2e6a444b7887de4d0b98812ea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:15.850458 13433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c061130f43d444248e147a2d54aeb5f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:15.850253 13577 tablet_service.cc:1467] Processing CreateTablet for tablet c061130f43d444248e147a2d54aeb5f1 (DEFAULT_TABLE table=test-table [id=cdab5c2e6a444b7887de4d0b98812ea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:15.856913 13577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c061130f43d444248e147a2d54aeb5f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:15.865826 13800 tablet_bootstrap.cc:492] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158: Bootstrap starting.
I20260924 03:12:15.873242 13800 tablet_bootstrap.cc:654] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:15.875260 13800 log.cc:824] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:15.879673 13802 tablet_bootstrap.cc:492] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0: Bootstrap starting.
I20260924 03:12:15.881982 13800 tablet_bootstrap.cc:492] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158: No bootstrap required, opened a new log
I20260924 03:12:15.882371 13800 ts_tablet_manager.cc:1397] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20260924 03:12:15.885442 13802 tablet_bootstrap.cc:654] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:15.886307 13803 tablet_bootstrap.cc:492] T c061130f43d444248e147a2d54aeb5f1 P 635d5ccfe07c477587d79700b550e7f7: Bootstrap starting.
I20260924 03:12:15.887806 13802 log.cc:824] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:15.890332 13803 tablet_bootstrap.cc:654] T c061130f43d444248e147a2d54aeb5f1 P 635d5ccfe07c477587d79700b550e7f7: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:15.891346 13803 log.cc:824] T c061130f43d444248e147a2d54aeb5f1 P 635d5ccfe07c477587d79700b550e7f7: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:15.891502 13802 tablet_bootstrap.cc:492] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0: No bootstrap required, opened a new log
I20260924 03:12:15.891799 13802 ts_tablet_manager.cc:1397] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.002s
I20260924 03:12:15.892979 13800 raft_consensus.cc:348] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0755c3a209642c78efe7641bcaf6158" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } } peers { permanent_uuid: "635d5ccfe07c477587d79700b550e7f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40209 } } peers { permanent_uuid: "caec79b7d075429f9118bc256ccaa3d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43147 } }
I20260924 03:12:15.893483 13800 raft_consensus.cc:374] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:15.893596 13800 raft_consensus.cc:729] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0755c3a209642c78efe7641bcaf6158, State: Initialized, Role: FOLLOWER
I20260924 03:12:15.894253 13800 consensus_queue.cc:260] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0755c3a209642c78efe7641bcaf6158" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } } peers { permanent_uuid: "635d5ccfe07c477587d79700b550e7f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40209 } } peers { permanent_uuid: "caec79b7d075429f9118bc256ccaa3d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43147 } }
I20260924 03:12:15.894487 13803 tablet_bootstrap.cc:492] T c061130f43d444248e147a2d54aeb5f1 P 635d5ccfe07c477587d79700b550e7f7: No bootstrap required, opened a new log
I20260924 03:12:15.894819 13803 ts_tablet_manager.cc:1397] T c061130f43d444248e147a2d54aeb5f1 P 635d5ccfe07c477587d79700b550e7f7: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260924 03:12:15.895458 13777 heartbeater.cc:503] Master 127.0.97.62:42023 was elected leader, sending a full tablet report...
I20260924 03:12:15.896246 13800 ts_tablet_manager.cc:1428] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260924 03:12:15.902181 13802 raft_consensus.cc:348] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0755c3a209642c78efe7641bcaf6158" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } } peers { permanent_uuid: "635d5ccfe07c477587d79700b550e7f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40209 } } peers { permanent_uuid: "caec79b7d075429f9118bc256ccaa3d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43147 } }
I20260924 03:12:15.902751 13802 raft_consensus.cc:374] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:15.902861 13802 raft_consensus.cc:729] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caec79b7d075429f9118bc256ccaa3d0, State: Initialized, Role: FOLLOWER
I20260924 03:12:15.903448 13802 consensus_queue.cc:260] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0755c3a209642c78efe7641bcaf6158" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } } peers { permanent_uuid: "635d5ccfe07c477587d79700b550e7f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40209 } } peers { permanent_uuid: "caec79b7d075429f9118bc256ccaa3d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43147 } }
I20260924 03:12:15.903833 13803 raft_consensus.cc:348] T c061130f43d444248e147a2d54aeb5f1 P 635d5ccfe07c477587d79700b550e7f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0755c3a209642c78efe7641bcaf6158" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } } peers { permanent_uuid: "635d5ccfe07c477587d79700b550e7f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40209 } } peers { permanent_uuid: "caec79b7d075429f9118bc256ccaa3d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43147 } }
I20260924 03:12:15.904209 13803 raft_consensus.cc:374] T c061130f43d444248e147a2d54aeb5f1 P 635d5ccfe07c477587d79700b550e7f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:15.904327 13803 raft_consensus.cc:729] T c061130f43d444248e147a2d54aeb5f1 P 635d5ccfe07c477587d79700b550e7f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635d5ccfe07c477587d79700b550e7f7, State: Initialized, Role: FOLLOWER
I20260924 03:12:15.904593 13498 heartbeater.cc:503] Master 127.0.97.62:42023 was elected leader, sending a full tablet report...
I20260924 03:12:15.904958 13803 consensus_queue.cc:260] T c061130f43d444248e147a2d54aeb5f1 P 635d5ccfe07c477587d79700b550e7f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0755c3a209642c78efe7641bcaf6158" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } } peers { permanent_uuid: "635d5ccfe07c477587d79700b550e7f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40209 } } peers { permanent_uuid: "caec79b7d075429f9118bc256ccaa3d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43147 } }
I20260924 03:12:15.905400 13802 ts_tablet_manager.cc:1428] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260924 03:12:15.906143 13643 heartbeater.cc:503] Master 127.0.97.62:42023 was elected leader, sending a full tablet report...
I20260924 03:12:15.906699 13803 ts_tablet_manager.cc:1428] T c061130f43d444248e147a2d54aeb5f1 P 635d5ccfe07c477587d79700b550e7f7: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260924 03:12:15.945926 13811 raft_consensus.cc:482] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:12:15.946187 13811 raft_consensus.cc:504] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0755c3a209642c78efe7641bcaf6158" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } } peers { permanent_uuid: "635d5ccfe07c477587d79700b550e7f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40209 } } peers { permanent_uuid: "caec79b7d075429f9118bc256ccaa3d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43147 } }
I20260924 03:12:15.947984 13811 leader_election.cc:288] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0755c3a209642c78efe7641bcaf6158 (127.0.97.3:45531), 635d5ccfe07c477587d79700b550e7f7 (127.0.97.2:40209)
I20260924 03:12:15.954605 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:15.948456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44539 (local address 127.0.97.3:45531)
0924 03:12:15.948717 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:15.948722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:15.949256 (+ 534us) server_negotiation.cc:407] Connection header received
0924 03:12:15.949382 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:15.949387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:15.949463 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:15.949559 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:15.950312 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.951058 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.952880 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.953650 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.953806 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:15.953925 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:15.954224 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 03:12:15.954356 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260924 03:12:15.954679 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:15.948318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45531 (local address 127.0.97.1:44539)
0924 03:12:15.949037 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:15.949055 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:15.949194 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:15.949759 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:15.949762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:15.949784 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:15.950151 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:15.950160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.951191 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.951199 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:15.952722 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:15.952735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.953871 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.953877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:15.954061 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:15.954084 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 03:12:15.954200 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 03:12:15.954338 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":349,"threads_started":1}
I20260924 03:12:15.955036 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:15.949465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49479 (local address 127.0.97.2:40209)
0924 03:12:15.949660 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:15.949665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:15.949982 (+ 317us) server_negotiation.cc:407] Connection header received
0924 03:12:15.950174 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:15.950179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:15.950255 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:15.950366 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:15.950868 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.951566 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.953293 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.954061 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.954220 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:15.954341 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:15.954613 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 03:12:15.954734 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:12:15.955087 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:15.949139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40209 (local address 127.0.97.1:49479)
0924 03:12:15.949868 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:15.949882 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:15.950032 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:15.950379 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:15.950383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:15.950405 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:15.950724 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:15.950731 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.951740 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.951744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:15.953149 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:15.953160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.954286 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.954292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:15.954451 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:15.954476 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 03:12:15.954576 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 03:12:15.954696 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":433,"threads_started":1}
I20260924 03:12:15.955868 13732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c061130f43d444248e147a2d54aeb5f1" candidate_uuid: "caec79b7d075429f9118bc256ccaa3d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0755c3a209642c78efe7641bcaf6158" is_pre_election: true
I20260924 03:12:15.956229 13597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c061130f43d444248e147a2d54aeb5f1" candidate_uuid: "caec79b7d075429f9118bc256ccaa3d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "635d5ccfe07c477587d79700b550e7f7" is_pre_election: true
I20260924 03:12:15.956373 13732 raft_consensus.cc:2454] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caec79b7d075429f9118bc256ccaa3d0 in term 0.
I20260924 03:12:15.956705 13597 raft_consensus.cc:2454] T c061130f43d444248e147a2d54aeb5f1 P 635d5ccfe07c477587d79700b550e7f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caec79b7d075429f9118bc256ccaa3d0 in term 0.
I20260924 03:12:15.957151 13385 leader_election.cc:302] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0755c3a209642c78efe7641bcaf6158, caec79b7d075429f9118bc256ccaa3d0; no voters:
I20260924 03:12:15.957535 13811 raft_consensus.cc:2781] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:12:15.957710 13811 raft_consensus.cc:482] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:12:15.957820 13811 raft_consensus.cc:3037] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:15.960120 13811 raft_consensus.cc:504] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0755c3a209642c78efe7641bcaf6158" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } } peers { permanent_uuid: "635d5ccfe07c477587d79700b550e7f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40209 } } peers { permanent_uuid: "caec79b7d075429f9118bc256ccaa3d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43147 } }
I20260924 03:12:15.960729 13811 leader_election.cc:288] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 [CANDIDATE]: Term 1 election: Requested vote from peers b0755c3a209642c78efe7641bcaf6158 (127.0.97.3:45531), 635d5ccfe07c477587d79700b550e7f7 (127.0.97.2:40209)
I20260924 03:12:15.961313 13597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c061130f43d444248e147a2d54aeb5f1" candidate_uuid: "caec79b7d075429f9118bc256ccaa3d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "635d5ccfe07c477587d79700b550e7f7"
I20260924 03:12:15.961315 13732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c061130f43d444248e147a2d54aeb5f1" candidate_uuid: "caec79b7d075429f9118bc256ccaa3d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0755c3a209642c78efe7641bcaf6158"
I20260924 03:12:15.961531 13597 raft_consensus.cc:3037] T c061130f43d444248e147a2d54aeb5f1 P 635d5ccfe07c477587d79700b550e7f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:15.961537 13732 raft_consensus.cc:3037] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158 [term 0 FOLLOWER]: Advancing to term 1
W20260924 03:12:15.963440 13778 tablet.cc:2369] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:12:15.963884 13597 raft_consensus.cc:2454] T c061130f43d444248e147a2d54aeb5f1 P 635d5ccfe07c477587d79700b550e7f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caec79b7d075429f9118bc256ccaa3d0 in term 1.
I20260924 03:12:15.964044 13732 raft_consensus.cc:2454] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caec79b7d075429f9118bc256ccaa3d0 in term 1.
I20260924 03:12:15.964383 13385 leader_election.cc:302] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 635d5ccfe07c477587d79700b550e7f7, caec79b7d075429f9118bc256ccaa3d0; no voters:
I20260924 03:12:15.964742 13811 raft_consensus.cc:2781] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:15.965211 13811 raft_consensus.cc:686] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 [term 1 LEADER]: Becoming Leader. State: Replica: caec79b7d075429f9118bc256ccaa3d0, State: Running, Role: LEADER
I20260924 03:12:15.965708 13811 consensus_queue.cc:237] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0755c3a209642c78efe7641bcaf6158" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } } peers { permanent_uuid: "635d5ccfe07c477587d79700b550e7f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40209 } } peers { permanent_uuid: "caec79b7d075429f9118bc256ccaa3d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43147 } }
I20260924 03:12:15.970567 13299 catalog_manager.cc:5495] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to caec79b7d075429f9118bc256ccaa3d0 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "caec79b7d075429f9118bc256ccaa3d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0755c3a209642c78efe7641bcaf6158" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "635d5ccfe07c477587d79700b550e7f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "caec79b7d075429f9118bc256ccaa3d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43147 } health_report { overall_health: HEALTHY } } }
I20260924 03:12:15.989813 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:15.983153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56476 (local address 127.0.97.1:43147)
0924 03:12:15.983527 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:15.983532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:15.983587 (+ 55us) server_negotiation.cc:407] Connection header received
0924 03:12:15.983862 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:15.983869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:15.983994 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:15.984145 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:12:15.984658 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.985456 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.987294 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:15.987692 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:15.987792 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:15.988320 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:12:15.988978 (+ 658us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:12:15.989097 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:15.989330 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 03:12:15.989398 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 03:12:15.997949 13732 raft_consensus.cc:1261] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158 [term 1 FOLLOWER]: Refusing update from remote peer caec79b7d075429f9118bc256ccaa3d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:15.997941 13597 raft_consensus.cc:1261] T c061130f43d444248e147a2d54aeb5f1 P 635d5ccfe07c477587d79700b550e7f7 [term 1 FOLLOWER]: Refusing update from remote peer caec79b7d075429f9118bc256ccaa3d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:15.998878 13815 consensus_queue.cc:1035] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0755c3a209642c78efe7641bcaf6158" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:15.999284 13811 consensus_queue.cc:1035] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "635d5ccfe07c477587d79700b550e7f7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:16.012219 13820 mvcc.cc:205] Tried to move back new op lower bound from 7332739219437912064 to 7332739219319271424. Current Snapshot: MvccSnapshot[applied={T|T < 7332739219437912064}]
I20260924 03:12:16.028878 13300 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:12:16.029258 13300 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:12:16.033123 13300 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82d0ed0de9a44703a9db9f28e4c05f52: Sending DeleteTablet for 3 replicas of tablet c061130f43d444248e147a2d54aeb5f1
I20260924 03:12:16.034428 13577 tablet_service.cc:1514] Processing DeleteTablet for tablet c061130f43d444248e147a2d54aeb5f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45748
I20260924 03:12:16.034428 13712 tablet_service.cc:1514] Processing DeleteTablet for tablet c061130f43d444248e147a2d54aeb5f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54880
I20260924 03:12:16.034467 13433 tablet_service.cc:1514] Processing DeleteTablet for tablet c061130f43d444248e147a2d54aeb5f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56464
I20260924 03:12:16.041467 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 13337
I20260924 03:12:16.049023 13828 tablet_replica.cc:330] T c061130f43d444248e147a2d54aeb5f1 P 635d5ccfe07c477587d79700b550e7f7: stopping tablet replica
I20260924 03:12:16.049633 13829 tablet_replica.cc:330] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158: stopping tablet replica
I20260924 03:12:16.051795 13828 raft_consensus.cc:2229] T c061130f43d444248e147a2d54aeb5f1 P 635d5ccfe07c477587d79700b550e7f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:12:16.052294 13829 raft_consensus.cc:2229] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:12:16.052932 13829 raft_consensus.cc:2258] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:16.053442 13828 raft_consensus.cc:2258] T c061130f43d444248e147a2d54aeb5f1 P 635d5ccfe07c477587d79700b550e7f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:16.054188 13830 tablet_replica.cc:330] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0: stopping tablet replica
I20260924 03:12:16.054629 13830 raft_consensus.cc:2229] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:12:16.055374 13830 raft_consensus.cc:2258] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:16.057830 13830 ts_tablet_manager.cc:1905] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:16.060444 13829 ts_tablet_manager.cc:1905] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:16.061949 13828 ts_tablet_manager.cc:1905] T c061130f43d444248e147a2d54aeb5f1 P 635d5ccfe07c477587d79700b550e7f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:16.064100 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.042667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56488 (local address 127.0.97.1:43147)
0924 03:12:16.042898 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:16.042906 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:16.043363 (+ 457us) server_negotiation.cc:407] Connection header received
0924 03:12:16.043578 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:16.043582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:16.043665 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:16.043773 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:16.046663 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.048085 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.050086 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.050949 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.051115 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.056411 (+ 5296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:16.056432 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:16.056452 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:16.056549 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:16.060821 (+ 4272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:16.061692 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:16.061700 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:16.061704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:16.061959 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:16.062384 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:16.062392 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:16.062399 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:16.062819 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:16.063126 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:16.063464 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 03:12:16.063704 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 03:12:16.064857 13493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:16.069617 13828 ts_tablet_manager.cc:1918] T c061130f43d444248e147a2d54aeb5f1 P 635d5ccfe07c477587d79700b550e7f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:16.069855 13828 log.cc:1196] T c061130f43d444248e147a2d54aeb5f1 P 635d5ccfe07c477587d79700b550e7f7: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/c061130f43d444248e147a2d54aeb5f1
I20260924 03:12:16.070480 13828 ts_tablet_manager.cc:1939] T c061130f43d444248e147a2d54aeb5f1 P 635d5ccfe07c477587d79700b550e7f7: Deleting consensus metadata
I20260924 03:12:16.072669 13284 catalog_manager.cc:4841] TS 635d5ccfe07c477587d79700b550e7f7 (127.0.97.2:40209): tablet c061130f43d444248e147a2d54aeb5f1 (table test-table [id=cdab5c2e6a444b7887de4d0b98812ea2]) successfully deleted
I20260924 03:12:16.073766 13829 ts_tablet_manager.cc:1918] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:16.073923 13829 log.cc:1196] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/c061130f43d444248e147a2d54aeb5f1
I20260924 03:12:16.074339 13829 ts_tablet_manager.cc:1939] T c061130f43d444248e147a2d54aeb5f1 P b0755c3a209642c78efe7641bcaf6158: Deleting consensus metadata
I20260924 03:12:16.075691 13284 catalog_manager.cc:4841] TS b0755c3a209642c78efe7641bcaf6158 (127.0.97.3:45531): tablet c061130f43d444248e147a2d54aeb5f1 (table test-table [id=cdab5c2e6a444b7887de4d0b98812ea2]) successfully deleted
I20260924 03:12:16.167979 13830 ts_tablet_manager.cc:1918] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:16.168244 13830 log.cc:1196] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/c061130f43d444248e147a2d54aeb5f1
I20260924 03:12:16.168704 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 13337
I20260924 03:12:16.169071 13830 ts_tablet_manager.cc:1939] T c061130f43d444248e147a2d54aeb5f1 P caec79b7d075429f9118bc256ccaa3d0: Deleting consensus metadata
I20260924 03:12:16.187554 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 13502
W20260924 03:12:16.187457 13286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43147 (error 108)
I20260924 03:12:16.189395 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.188931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43147 (local address 127.0.0.1:44044)
0924 03:12:16.189122 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:16.189242 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 03:12:16.189647 13286 catalog_manager.cc:4584] TS caec79b7d075429f9118bc256ccaa3d0 (127.0.97.1:43147): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c061130f43d444248e147a2d54aeb5f1: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43147: connect: Connection refused (error 111)
I20260924 03:12:16.205041 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.188420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45762 (local address 127.0.97.2:40209)
0924 03:12:16.188660 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:16.188667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:16.188766 (+ 99us) server_negotiation.cc:407] Connection header received
0924 03:12:16.188985 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:16.188989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:16.189062 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:16.189172 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:16.190707 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.191465 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.192922 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.193493 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.193641 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.196199 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:16.196224 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:16.196239 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:16.196275 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:16.200762 (+ 4487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:16.201840 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:16.201851 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:16.201857 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:16.202035 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:16.202605 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:16.202611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:16.202615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:16.203097 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:16.203417 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:16.204553 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0924 03:12:16.204746 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260924 03:12:16.205713 13637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:16.243508 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.242995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43147 (local address 127.0.0.1:44056)
0924 03:12:16.243218 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:16.243345 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 03:12:16.244654 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.244242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43147 (local address 127.0.0.1:44064)
0924 03:12:16.244387 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:16.244461 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 03:12:16.300611 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 13502
I20260924 03:12:16.305337 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.304717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43147 (local address 127.0.0.1:44078)
0924 03:12:16.304983 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:16.305101 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 03:12:16.306496 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.306064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43147 (local address 127.0.0.1:44090)
0924 03:12:16.306243 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:16.306346 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 03:12:16.318200 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 13647
I20260924 03:12:16.332731 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.319177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52356 (local address 127.0.97.3:45531)
0924 03:12:16.319370 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:16.319377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:16.319402 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:12:16.319468 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:16.319473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:16.319549 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:16.319660 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:16.321092 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.321833 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.323144 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.323577 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.323719 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.326612 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:16.326628 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:16.326644 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:16.326699 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:16.329739 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:16.330426 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:16.330437 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:16.330446 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:16.330566 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:16.330955 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:16.330962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:16.330965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:16.331328 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:16.331656 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:16.332063 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 03:12:16.332287 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:12:16.333400 13772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:16.387320 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.386775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43147 (local address 127.0.0.1:44096)
0924 03:12:16.387027 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:16.387141 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260924 03:12:16.388675 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.388151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43147 (local address 127.0.0.1:44098)
0924 03:12:16.388363 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:16.388435 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260924 03:12:16.426457 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 13647
I20260924 03:12:16.442931 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 13267
I20260924 03:12:16.444262 13330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:16.559027 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 13267
I20260924 03:12:16.584036 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 03:12:16.930107 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 03:12:16.930513 388 mini_ranger.cc:66] Stopping Ranger...
I20260924 03:12:17.276293 388 mini_ranger.cc:68] Stopped Ranger
2026-09-24 03:12:17.276 UTC [12319] LOG: received smart shutdown request
2026-09-24 03:12:17.281 UTC [12319] LOG: background worker "logical replication launcher" (PID 12328) exited with exit code 1
2026-09-24 03:12:17.281 UTC [12323] LOG: shutting down
2026-09-24 03:12:17.505 UTC [12319] LOG: database system is shut down
2026-09-24T03:12:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55372 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:17 dist-test-slave-5l98 krb5kdc[13844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:17 dist-test-slave-5l98 krb5kdc[13844](info): set up 2 sockets
Sep 24 03:12:17 dist-test-slave-5l98 krb5kdc[13844](info): commencing operation
krb5kdc: starting...
W20260924 03:12:18.797083 388 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:12:18 dist-test-slave-5l98 krb5kdc[13844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:12:19.041446 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38027
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38027,127.0.97.61:38257,127.0.97.60:44007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:19.374792 13857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:19.375094 13857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:19.375262 13857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:19.383388 13857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:19.383519 13857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:19.383656 13857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:19.383742 13857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:19.383813 13857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:19.394177 13857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38027,127.0.97.61:38257,127.0.97.60:44007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:19.395608 13857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:19.397087 13857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:19.410897 13862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:19.410868 13865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:19.410917 13863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:19.411929 13857 server_base.cc:1034] running on GCE node
I20260924 03:12:19.412886 13857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:19.414382 13857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:19.415601 13857 hybrid_clock.cc:648] HybridClock initialized: now 1790219539415543 us; error 51 us; skew 500 ppm
I20260924 03:12:19.415957 13857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:19 dist-test-slave-5l98 krb5kdc[13844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:19.419356 13857 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:12:19.421036 13857 webserver.cc:460] Webserver started at http://127.0.97.62:35399/ using document root <none> and password file <none>
I20260924 03:12:19.421728 13857 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:19.421872 13857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:19.422202 13857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:19.425300 13857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "7182bef087f54bcdbaeb47017d0283b6"
format_stamp: "Formatted at 2026-09-24 03:12:19 on dist-test-slave-5l98"
server_key: "742093c26b3c65fa053cd4e8f2ef184c"
server_key_iv: "63d84d7c0c37978d1c3b3862a504ba0e"
server_key_version: "encryptionkey@0"
I20260924 03:12:19.426102 13857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "7182bef087f54bcdbaeb47017d0283b6"
format_stamp: "Formatted at 2026-09-24 03:12:19 on dist-test-slave-5l98"
server_key: "742093c26b3c65fa053cd4e8f2ef184c"
server_key_iv: "63d84d7c0c37978d1c3b3862a504ba0e"
server_key_version: "encryptionkey@0"
I20260924 03:12:19.431433 13857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:12:19.435151 13872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:19.436622 13857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 03:12:19.436808 13857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "7182bef087f54bcdbaeb47017d0283b6"
format_stamp: "Formatted at 2026-09-24 03:12:19 on dist-test-slave-5l98"
server_key: "742093c26b3c65fa053cd4e8f2ef184c"
server_key_iv: "63d84d7c0c37978d1c3b3862a504ba0e"
server_key_version: "encryptionkey@0"
I20260924 03:12:19.437032 13857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:19.457525 13857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:19.462064 13857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:19.462360 13857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:19.481910 13857 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38027
I20260924 03:12:19.481948 13923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38027 every 8 connection(s)
I20260924 03:12:19.483276 13857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:19.484349 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 13857
I20260924 03:12:19.484632 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:12:19.485495 388 external_mini_cluster.cc:1428] Setting key 5e0ab9e841164fd02f16fec2d8c53266
I20260924 03:12:19.488056 13924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:19.496749 13924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 24 03:12:19 dist-test-slave-5l98 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:12:19.519996 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:19.500100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38027 (local address 127.0.0.1:48578)
0924 03:12:19.501323 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:19.501356 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:19.503194 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:19.504083 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:19.504090 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:19.504824 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:19.505146 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:19.505163 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:19.506675 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:19.506680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:19.507803 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:19.507814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:19.508845 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:19.508851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:19.509036 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:19.509723 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:19.509745 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:19.512231 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:19.515657 (+ 3426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:19.515669 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:19.515682 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:19.516082 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:19.516554 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:19.516558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:19.516561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:19.516708 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:19.517488 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:19.517497 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:19.517705 (+ 208us) client_negotiation.cc:769] Sending connection context
0924 03:12:19.517956 (+ 251us) client_negotiation.cc:240] Negotiation successful
0924 03:12:19.518647 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"mutex_wait_us":168,"spinlock_wait_cycles":16128,"thread_start_us":819,"threads_started":1}
I20260924 03:12:19.520035 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:19.500100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48578 (local address 127.0.97.62:38027)
0924 03:12:19.501032 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:19.501053 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:19.501526 (+ 473us) server_negotiation.cc:407] Connection header received
0924 03:12:19.503566 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:19.503575 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:19.503857 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:19.504068 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:19.505344 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:19.506518 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:19.507958 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:19.508613 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:19.508805 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:19.512534 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:19.512549 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:19.512565 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:19.512611 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:19.515495 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:19.516227 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:19.516236 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:19.516240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:19.516401 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:19.516815 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:19.516823 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:19.516826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:19.517327 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:19.517536 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:19.518104 (+ 568us) server_negotiation.cc:299] Negotiation successful
0924 03:12:19.518538 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":687,"thread_start_us":480,"threads_started":1}
I20260924 03:12:19.523332 13924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:12:19.524464 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:19.524028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38257 (local address 127.0.0.1:55876)
0924 03:12:19.524214 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:19.524338 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 03:12:19.524821 13875 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38257: connect: Connection refused (error 111)
I20260924 03:12:19.526518 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:19.526153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38257 (local address 127.0.0.1:55884)
0924 03:12:19.526302 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:19.526385 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 03:12:19.526780 13924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38257: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38257: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260924 03:12:19.539976 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38257
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38027,127.0.97.61:38257,127.0.97.60:44007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:12:19.576215 13924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } attempt: 1
I20260924 03:12:19.577733 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:19.577181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38257 (local address 127.0.0.1:55894)
0924 03:12:19.577425 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:19.577568 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 03:12:19.578898 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:19.578453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38257 (local address 127.0.0.1:55902)
0924 03:12:19.578617 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:19.578704 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 03:12:19.579123 13924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38257: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38257: connect: Connection refused (error 111)
I20260924 03:12:19.647415 13924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } attempt: 2
I20260924 03:12:19.648787 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:19.648202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38257 (local address 127.0.0.1:55916)
0924 03:12:19.648460 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:19.648614 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 03:12:19.650067 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:19.649619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38257 (local address 127.0.0.1:55930)
0924 03:12:19.649804 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:19.649903 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 03:12:19.650313 13924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38257: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38257: connect: Connection refused (error 111)
I20260924 03:12:19.741621 13924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } attempt: 3
I20260924 03:12:19.742837 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:19.742320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38257 (local address 127.0.0.1:55944)
0924 03:12:19.742552 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:19.742665 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 03:12:19.744093 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:19.743643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38257 (local address 127.0.0.1:55948)
0924 03:12:19.743808 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:19.743905 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 03:12:19.744320 13924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38257: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38257: connect: Connection refused (error 111)
W20260924 03:12:19.875061 13931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:19.875535 13931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:19.875649 13931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:19.883230 13931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:19.883345 13931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:19.883399 13931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:19.883459 13931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:19.883497 13931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:19.887606 13924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } attempt: 4
I20260924 03:12:19.889178 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:19.888614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38257 (local address 127.0.0.1:55962)
0924 03:12:19.888941 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:19.889042 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260924 03:12:19.890300 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:19.889881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38257 (local address 127.0.0.1:55966)
0924 03:12:19.890051 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:19.890145 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 03:12:19.890511 13924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38257: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38257: connect: Connection refused (error 111)
I20260924 03:12:19.894299 13931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38027,127.0.97.61:38257,127.0.97.60:44007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:19.895711 13931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:19.897114 13931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:19.910068 13936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:19.910014 13937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:19.909956 13939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:19.910843 13931 server_base.cc:1034] running on GCE node
I20260924 03:12:19.911661 13931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:19.913275 13931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:19.914521 13931 hybrid_clock.cc:648] HybridClock initialized: now 1790219539914459 us; error 50 us; skew 500 ppm
I20260924 03:12:19.914911 13931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:19 dist-test-slave-5l98 krb5kdc[13844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:19.918263 13931 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260924 03:12:19.919857 13931 webserver.cc:460] Webserver started at http://127.0.97.61:36061/ using document root <none> and password file <none>
I20260924 03:12:19.920598 13931 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:19.920706 13931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:19.921036 13931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:19.923983 13931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-1/data/instance:
uuid: "841eb317eab2452989294b18ebc84950"
format_stamp: "Formatted at 2026-09-24 03:12:19 on dist-test-slave-5l98"
server_key: "377ce1bead736215aa6af6ef3b6253cc"
server_key_iv: "ce8eb15c76908a7f401d003f3b89b22a"
server_key_version: "encryptionkey@0"
I20260924 03:12:19.924818 13931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-1/wal/instance:
uuid: "841eb317eab2452989294b18ebc84950"
format_stamp: "Formatted at 2026-09-24 03:12:19 on dist-test-slave-5l98"
server_key: "377ce1bead736215aa6af6ef3b6253cc"
server_key_iv: "ce8eb15c76908a7f401d003f3b89b22a"
server_key_version: "encryptionkey@0"
I20260924 03:12:19.930218 13931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 03:12:19.934065 13946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:19.935671 13931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:12:19.935823 13931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-1/wal
uuid: "841eb317eab2452989294b18ebc84950"
format_stamp: "Formatted at 2026-09-24 03:12:19 on dist-test-slave-5l98"
server_key: "377ce1bead736215aa6af6ef3b6253cc"
server_key_iv: "ce8eb15c76908a7f401d003f3b89b22a"
server_key_version: "encryptionkey@0"
I20260924 03:12:19.936117 13931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:19.959300 13931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:19.963737 13931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:19.964049 13931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:19.983506 13931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38257
I20260924 03:12:19.983594 13997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38257 every 8 connection(s)
I20260924 03:12:19.985008 13931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-1/data/info.pb
I20260924 03:12:19.988853 13998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:19.992405 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 13931
I20260924 03:12:19.992681 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-1/wal/instance
I20260924 03:12:19.993124 388 external_mini_cluster.cc:1428] Setting key 1d56cb948759483f8040dcc5114879e6
I20260924 03:12:19.995421 13998 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:12:20 dist-test-slave-5l98 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260924 03:12:20.019550 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:19.998162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48580 (local address 127.0.97.62:38027)
0924 03:12:19.998347 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:19.998352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:19.999893 (+ 1541us) server_negotiation.cc:407] Connection header received
0924 03:12:20.002234 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:20.002243 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:20.002340 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:20.002463 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:20.004096 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.005270 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.007129 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.007916 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.008067 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.012348 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:20.012354 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:20.012360 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:20.012410 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:20.015713 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.016658 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.016662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.016665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.016787 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.017293 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.017303 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.017309 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.017894 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:20.018031 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:20.019098 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0924 03:12:20.019280 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:12:20.021183 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:19.998757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38027 (local address 127.0.0.1:48580)
0924 03:12:19.999679 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:19.999733 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:20.001920 (+ 2187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:20.002742 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:20.002751 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:20.003322 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:20.003887 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:20.003906 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.005570 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.005576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:20.006957 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:20.006972 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.008187 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.008193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:20.008401 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.009357 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:20.009400 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:20.012063 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:20.015877 (+ 3814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:20.015885 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:20.015900 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:20.016414 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:20.016889 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:20.016893 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:20.016895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:20.017089 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:20.018086 (+ 997us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:20.018094 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:20.018636 (+ 542us) client_negotiation.cc:769] Sending connection context
0924 03:12:20.018968 (+ 332us) client_negotiation.cc:240] Negotiation successful
0924 03:12:20.019613 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":394,"threads_started":1}
I20260924 03:12:20.023268 13998 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:12:20 dist-test-slave-5l98 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 03:12:20.039904 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.024001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38257 (local address 127.0.0.1:55974)
0924 03:12:20.024214 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.024235 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:20.024381 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:20.025549 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:20.025552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:20.025914 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:20.026068 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:20.026076 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.027633 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.027641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:20.029253 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:20.029270 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.030137 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.030146 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:20.030402 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.031003 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:20.031030 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:20.033623 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:20.037316 (+ 3693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:20.037323 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:20.037327 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:20.037695 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:20.038080 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:20.038086 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:20.038089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:20.038228 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:20.038831 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:20.038838 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:20.039017 (+ 179us) client_negotiation.cc:769] Sending connection context
0924 03:12:20.039184 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 03:12:20.039462 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 03:12:20.039943 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.024259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55974 (local address 127.0.97.61:38257)
0924 03:12:20.025041 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:20.025046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:20.025072 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:12:20.025116 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:20.025122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:20.025363 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:20.025522 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:20.026240 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.027471 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.029416 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.029955 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.030068 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.033869 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:20.033880 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:20.033921 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:20.033964 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:20.037178 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.037830 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.037845 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.037848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.037965 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.038340 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.038347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.038351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.038704 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:20.038933 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:20.039347 (+ 414us) server_negotiation.cc:299] Negotiation successful
0924 03:12:20.039565 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":476,"threads_started":1}
I20260924 03:12:20.042800 13998 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:12:20.044076 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.043569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44007 (local address 127.0.0.1:51390)
0924 03:12:20.043753 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.043879 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260924 03:12:20.044385 13947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
I20260924 03:12:20.046048 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.045732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44007 (local address 127.0.0.1:51392)
0924 03:12:20.045862 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.045937 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 03:12:20.046294 13998 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44007: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260924 03:12:20.049315 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44007
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38027,127.0.97.61:38257,127.0.97.60:44007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:12:20.095683 13998 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } attempt: 1
I20260924 03:12:20.097084 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.096428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44007 (local address 127.0.0.1:51400)
0924 03:12:20.096703 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.096837 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 03:12:20.098342 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.097821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44007 (local address 127.0.0.1:51410)
0924 03:12:20.098069 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.098169 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 03:12:20.098640 13998 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44007: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
I20260924 03:12:20.166885 13998 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } attempt: 2
I20260924 03:12:20.168185 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.167677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44007 (local address 127.0.0.1:51424)
0924 03:12:20.167930 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.168019 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260924 03:12:20.169291 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.168935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44007 (local address 127.0.0.1:51432)
0924 03:12:20.169096 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.169174 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 03:12:20.169509 13998 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44007: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
I20260924 03:12:20.189824 13924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } attempt: 5
Sep 24 03:12:20 dist-test-slave-5l98 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 03:12:20.207201 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.190802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.0.97.61:38257)
0924 03:12:20.191014 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:20.191021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:20.191045 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:12:20.191140 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:20.191146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:20.191249 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:20.191367 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:20.192821 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.194166 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.196273 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.197086 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.197888 (+ 802us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.201124 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:20.201132 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:20.201141 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:20.201205 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:20.204433 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.205191 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.205197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.205201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.205337 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.205758 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.205764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.205767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.206063 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:20.206222 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:20.206638 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 03:12:20.206867 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 03:12:20.207207 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.190667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38257 (local address 127.0.0.1:55980)
0924 03:12:20.190898 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.190924 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:20.191039 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:20.191452 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:20.191457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:20.192011 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:20.192569 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:20.192592 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.194361 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.194371 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:20.196068 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:20.196097 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.197277 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.197283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:20.197507 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.198225 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:20.198254 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:20.200904 (+ 2650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:20.204641 (+ 3737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:20.204647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:20.204652 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:20.205057 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:20.205468 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:20.205473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:20.205476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:20.205640 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:20.206197 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:20.206202 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:20.206384 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 03:12:20.206531 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 03:12:20.206839 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 03:12:20.208979 13924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:12:20.210143 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.209721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44007 (local address 127.0.0.1:51444)
0924 03:12:20.209914 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.209993 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 03:12:20.211117 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.210762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44007 (local address 127.0.0.1:51448)
0924 03:12:20.210944 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.211020 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 03:12:20.211349 13924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44007: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
I20260924 03:12:20.260767 13998 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } attempt: 3
I20260924 03:12:20.261982 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.261498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44007 (local address 127.0.0.1:51462)
0924 03:12:20.261729 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.261822 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 03:12:20.262674 13924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } attempt: 1
I20260924 03:12:20.263429 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.262864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44007 (local address 127.0.0.1:51470)
0924 03:12:20.263182 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.263286 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
W20260924 03:12:20.263676 13998 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44007: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
I20260924 03:12:20.263737 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.263252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44007 (local address 127.0.0.1:51484)
0924 03:12:20.263440 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.263551 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 03:12:20.264853 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.264443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44007 (local address 127.0.0.1:51494)
0924 03:12:20.264656 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.264736 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260924 03:12:20.265090 13924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44007: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
I20260924 03:12:20.333343 13924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } attempt: 2
I20260924 03:12:20.334578 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.334078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44007 (local address 127.0.0.1:51506)
0924 03:12:20.334318 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.334413 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 03:12:20.335685 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.335319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44007 (local address 127.0.0.1:51508)
0924 03:12:20.335504 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.335576 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260924 03:12:20.335920 13924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44007: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
W20260924 03:12:20.392639 14005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:20.393014 14005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:20.393178 14005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:20.400691 14005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:20.400840 14005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:20.400959 14005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:20.401055 14005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:20.401142 14005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:20.406981 13998 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } attempt: 4
I20260924 03:12:20.408267 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.407740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44007 (local address 127.0.0.1:51512)
0924 03:12:20.408012 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.408109 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260924 03:12:20.409567 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.409042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44007 (local address 127.0.0.1:51522)
0924 03:12:20.409256 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.409447 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260924 03:12:20.409806 13998 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44007: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
I20260924 03:12:20.412344 14005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38027,127.0.97.61:38257,127.0.97.60:44007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:20.413830 14005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:20.415205 14005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:20.427708 14010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:20.427639 14013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:20.427677 14011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:20.428359 14005 server_base.cc:1034] running on GCE node
I20260924 03:12:20.429346 14005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:20.430903 14005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:20.432181 14005 hybrid_clock.cc:648] HybridClock initialized: now 1790219540432125 us; error 45 us; skew 500 ppm
I20260924 03:12:20.432760 14005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:20 dist-test-slave-5l98 krb5kdc[13844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:20.436142 14005 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260924 03:12:20.438004 14005 webserver.cc:460] Webserver started at http://127.0.97.60:37487/ using document root <none> and password file <none>
I20260924 03:12:20.438783 14005 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:20.438954 14005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:20.439318 14005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:20.442164 13924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } attempt: 3
I20260924 03:12:20.443329 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.442838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44007 (local address 127.0.0.1:51526)
0924 03:12:20.443083 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.443179 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 03:12:20.443212 14005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-2/data/instance:
uuid: "5f7b370b8bf54a33992fa9c3076db576"
format_stamp: "Formatted at 2026-09-24 03:12:20 on dist-test-slave-5l98"
server_key: "1546bc3e1eed287a55d6b68d3df2f356"
server_key_iv: "c01143695fda3499ed9edcadf4223d1c"
server_key_version: "encryptionkey@0"
I20260924 03:12:20.444222 14005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-2/wal/instance:
uuid: "5f7b370b8bf54a33992fa9c3076db576"
format_stamp: "Formatted at 2026-09-24 03:12:20 on dist-test-slave-5l98"
server_key: "1546bc3e1eed287a55d6b68d3df2f356"
server_key_iv: "c01143695fda3499ed9edcadf4223d1c"
server_key_version: "encryptionkey@0"
I20260924 03:12:20.444669 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.444054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44007 (local address 127.0.0.1:51528)
0924 03:12:20.444240 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.444324 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260924 03:12:20.444927 13924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44007: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44007: connect: Connection refused (error 111)
I20260924 03:12:20.450436 14005 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 03:12:20.454522 14020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:20.456199 14005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:12:20.456424 14005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-2/wal
uuid: "5f7b370b8bf54a33992fa9c3076db576"
format_stamp: "Formatted at 2026-09-24 03:12:20 on dist-test-slave-5l98"
server_key: "1546bc3e1eed287a55d6b68d3df2f356"
server_key_iv: "c01143695fda3499ed9edcadf4223d1c"
server_key_version: "encryptionkey@0"
I20260924 03:12:20.456800 14005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:20.477922 14005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:20.482380 14005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:20.482698 14005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:20.502228 14005 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44007
I20260924 03:12:20.502223 14071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44007 every 8 connection(s)
I20260924 03:12:20.503703 14005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-2/data/info.pb
I20260924 03:12:20.507383 14072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:20.512428 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 14005
I20260924 03:12:20.512765 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219327163871-388-0/minicluster-data/master-2/wal/instance
I20260924 03:12:20.513202 388 external_mini_cluster.cc:1428] Setting key 3f6c961434c702507ffc9ca717d8d97c
I20260924 03:12:20.514071 14072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:12:20 dist-test-slave-5l98 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 24 03:12:20 dist-test-slave-5l98 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 03:12:20.533890 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.517068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48596 (local address 127.0.97.62:38027)
0924 03:12:20.517252 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:20.517256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:20.517895 (+ 639us) server_negotiation.cc:407] Connection header received
0924 03:12:20.518678 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:20.518683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:20.518775 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:20.518878 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:20.520221 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.521027 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.522743 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.523425 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.523546 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.527826 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:20.527831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:20.527835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:20.527869 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:20.530773 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.531553 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.531561 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.531566 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.531707 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.532155 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.532160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.532163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.532417 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:20.532606 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:20.533297 (+ 691us) server_negotiation.cc:299] Negotiation successful
0924 03:12:20.533514 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:12:20.535211 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.516995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38027 (local address 127.0.0.1:48596)
0924 03:12:20.517745 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.517781 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:20.518384 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:20.518976 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:20.518982 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:20.519666 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:20.520033 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:20.520049 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.521189 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.521194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:20.522601 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:20.522612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.523704 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.523710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:20.524102 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.524997 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:20.525022 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:20.527604 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:20.530920 (+ 3316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:20.530932 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:20.530945 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:20.531413 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:20.531823 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:20.531828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:20.531832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:20.532031 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:20.532585 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:20.532593 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:20.532924 (+ 331us) client_negotiation.cc:769] Sending connection context
0924 03:12:20.533209 (+ 285us) client_negotiation.cc:240] Negotiation successful
0924 03:12:20.533711 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":50,"thread_start_us":404,"threads_started":1}
I20260924 03:12:20.535279 14075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.515109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51534 (local address 127.0.97.60:44007)
0924 03:12:20.515964 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:20.515974 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:20.516015 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:12:20.518367 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:20.518387 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:20.518617 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:20.518796 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:20.520273 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.521798 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.523495 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.524436 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.524578 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.529554 (+ 4976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:20.529564 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:20.529578 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:20.529620 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:20.532426 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.533176 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.533184 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.533187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.533325 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.533712 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.533721 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.533723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.534105 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:20.534369 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:20.534718 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 03:12:20.534899 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":650,"thread_start_us":382,"threads_started":1}
I20260924 03:12:20.536959 14072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:12:20 dist-test-slave-5l98 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 24 03:12:20 dist-test-slave-5l98 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:12:20.551774 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.537595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38257 (local address 127.0.0.1:55996)
0924 03:12:20.538023 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.538043 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:20.538152 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:20.538623 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:20.538630 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:20.539127 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:20.539331 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:20.539344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.540640 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.540644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:20.541704 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:20.541718 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.542542 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.542549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:20.542774 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.543437 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:20.543461 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:20.545990 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:20.549348 (+ 3358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:20.549356 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:20.549359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:20.549751 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:20.550122 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:20.550128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:20.550135 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:20.550280 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:20.550770 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:20.550775 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:20.550925 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 03:12:20.551078 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 03:12:20.551358 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311}
I20260924 03:12:20.551795 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.537820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55996 (local address 127.0.97.61:38257)
0924 03:12:20.537995 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:20.537999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:20.538120 (+ 121us) server_negotiation.cc:407] Connection header received
0924 03:12:20.538266 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:20.538272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:20.538353 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:20.538449 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:20.539575 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.540437 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.541914 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.542380 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.542495 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.546213 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:20.546219 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:20.546222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:20.546258 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:20.549184 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.549878 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.549885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.549888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.550010 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.550388 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.550395 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.550399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.550655 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:20.550814 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:20.551173 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 03:12:20.551377 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:12:20.552809 14072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:12:20.555339 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.538547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48606 (local address 127.0.97.62:38027)
0924 03:12:20.538695 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:20.538698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:20.538713 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:12:20.538827 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:20.538830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:20.538879 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:20.538952 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:20.540396 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.541409 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.542831 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.543342 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.543446 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.548876 (+ 5430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:20.548889 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:20.548893 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:20.548947 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:20.551930 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.552824 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.552832 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.552836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.553002 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.553501 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.553507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.553510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.553873 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:20.554189 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:20.554711 (+ 522us) server_negotiation.cc:299] Negotiation successful
0924 03:12:20.554970 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 03:12:20 dist-test-slave-5l98 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 24 03:12:20 dist-test-slave-5l98 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 03:12:20.569628 14075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.553734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51542 (local address 127.0.97.60:44007)
0924 03:12:20.553922 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:20.553929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:20.554176 (+ 247us) server_negotiation.cc:407] Connection header received
0924 03:12:20.554307 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:20.554312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:20.554390 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:20.554553 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:20.555615 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.556684 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.558454 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.558858 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.558953 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.562803 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:20.562809 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:20.562812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:20.562881 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:20.566163 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.567023 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.567030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.567036 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.567215 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.567775 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.567779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.567782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.568147 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:20.568269 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:20.568907 (+ 638us) server_negotiation.cc:299] Negotiation successful
0924 03:12:20.569334 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":158,"server-negotiator.queue_time_us":67}
I20260924 03:12:20.569640 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.553734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44007 (local address 127.0.0.1:51542)
0924 03:12:20.554067 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.554086 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:20.554189 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:20.554631 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:20.554637 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:20.555154 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:20.555335 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:20.555349 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.556818 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.556824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:20.558297 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:20.558313 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.559020 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.559029 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:20.559251 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.560016 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:20.560033 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:20.562583 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:20.566333 (+ 3750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:20.566337 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:20.566340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:20.566879 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:20.567357 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:20.567361 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:20.567363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:20.567632 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:20.568292 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:20.568301 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:20.568473 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 03:12:20.568693 (+ 220us) client_negotiation.cc:240] Negotiation successful
0924 03:12:20.569112 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260924 03:12:20.574558 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.557250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56006 (local address 127.0.97.61:38257)
0924 03:12:20.557418 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:20.557425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:20.557603 (+ 178us) server_negotiation.cc:407] Connection header received
0924 03:12:20.557833 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:20.557837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:20.557905 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:20.557994 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:20.559707 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.560749 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.562349 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.562768 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.562918 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.567720 (+ 4802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:20.567728 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:20.567733 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:20.567790 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:20.571511 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.572285 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.572292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.572296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.572461 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.572979 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.572985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.572988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.573282 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:20.573598 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:20.573936 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 03:12:20.574188 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 03:12:20.575002 14072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576: Bootstrap starting.
I20260924 03:12:20.578790 14072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:20.579481 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:20.579918 14072 log.cc:824] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:20.582566 14072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576: No bootstrap required, opened a new log
I20260924 03:12:20.590555 14072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182bef087f54bcdbaeb47017d0283b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } } peers { permanent_uuid: "841eb317eab2452989294b18ebc84950" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } } peers { permanent_uuid: "5f7b370b8bf54a33992fa9c3076db576" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } }
I20260924 03:12:20.591145 14072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:20.591317 14072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f7b370b8bf54a33992fa9c3076db576, State: Initialized, Role: FOLLOWER
I20260924 03:12:20.592170 14072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182bef087f54bcdbaeb47017d0283b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } } peers { permanent_uuid: "841eb317eab2452989294b18ebc84950" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } } peers { permanent_uuid: "5f7b370b8bf54a33992fa9c3076db576" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } }
I20260924 03:12:20.598407 14085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182bef087f54bcdbaeb47017d0283b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } } peers { permanent_uuid: "841eb317eab2452989294b18ebc84950" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } } peers { permanent_uuid: "5f7b370b8bf54a33992fa9c3076db576" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } } }
I20260924 03:12:20.599484 14085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:12:20.600186 14072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:20.611717 14075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.584702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51558 (local address 127.0.97.60:44007)
0924 03:12:20.584938 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:20.584946 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:20.593189 (+ 8243us) server_negotiation.cc:407] Connection header received
0924 03:12:20.593434 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:20.593441 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:20.593541 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:20.593635 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:20.596466 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.597610 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.599117 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.599858 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.599950 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.604265 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:20.604272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:20.604279 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:20.604321 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:20.608301 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.609186 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.609194 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.609200 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.609379 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.609873 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.609878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.609881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.610314 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:20.610547 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:20.611101 (+ 554us) server_negotiation.cc:299] Negotiation successful
0924 03:12:20.611312 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 03:12:20.612089 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.584336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48612 (local address 127.0.97.62:38027)
0924 03:12:20.584586 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:20.584593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:20.589144 (+ 4551us) server_negotiation.cc:407] Connection header received
0924 03:12:20.589470 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:20.589476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:20.589558 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:20.589646 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:20.592082 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.593191 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.596352 (+ 3161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.597337 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.597474 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.602923 (+ 5449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:20.602933 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:20.602940 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:20.602992 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:20.607271 (+ 4279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.608332 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.608341 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.608347 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.608600 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.609167 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.609175 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.609180 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.609687 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:20.609995 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:20.611271 (+ 1276us) server_negotiation.cc:299] Negotiation successful
0924 03:12:20.611597 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:12:20.612970 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.589210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.0.97.61:38257)
0924 03:12:20.590551 (+ 1341us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:20.590561 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:20.593526 (+ 2965us) server_negotiation.cc:407] Connection header received
0924 03:12:20.593775 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:20.593782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:20.593851 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:20.593923 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:20.596466 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.597629 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.601003 (+ 3374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.601433 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.601585 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.605598 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:20.605607 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:20.605612 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:20.605656 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:20.609443 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.610357 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.610364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.610370 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.610517 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.610983 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.610991 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.610997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.611370 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:20.611615 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:20.612367 (+ 752us) server_negotiation.cc:299] Negotiation successful
0924 03:12:20.612646 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1172}
I20260924 03:12:20.622141 13924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } attempt: 4
W20260924 03:12:20.623024 14096 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:12:20.623229 14096 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:12:20.620075 14005 master_runner.cc:186] Error getting master registration for 127.0.97.62:38027: OK, instance_id { permanent_uuid: "7182bef087f54bcdbaeb47017d0283b6" instance_seqno: 1790219539476230 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:12:20.630379 14005 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 24 03:12:20 dist-test-slave-5l98 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 03:12:20.639110 14075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.623536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.0.97.60:44007)
0924 03:12:20.623736 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:20.623747 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:20.623766 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:12:20.623824 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:20.623829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:20.623945 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:20.624060 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:20.625026 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.626311 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.628191 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.628934 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.629677 (+ 743us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.632802 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:20.632809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:20.632813 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:20.632848 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:20.636399 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.637263 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.637269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.637277 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.637429 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.637836 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.637839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.637841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.638100 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:20.638254 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:20.638687 (+ 433us) server_negotiation.cc:299] Negotiation successful
0924 03:12:20.638854 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:12:20.639493 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.622697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44007 (local address 127.0.0.1:51560)
0924 03:12:20.623048 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.623069 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:20.623175 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:20.624115 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:20.624120 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:20.624654 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:20.624865 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:20.624878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.626486 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.626492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:20.627993 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:20.628010 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.629103 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.629110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:20.629324 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.630066 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:20.630090 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:20.632622 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:20.636622 (+ 4000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:20.636629 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:20.636635 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:20.637137 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:20.637547 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:20.637552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:20.637556 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:20.637725 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:20.638268 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:20.638273 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:20.638438 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 03:12:20.638606 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 03:12:20.638921 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230}
I20260924 03:12:20.644544 13924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6: Bootstrap starting.
I20260924 03:12:20.647826 13924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:20.648957 13924 log.cc:824] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:20.651618 13924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6: No bootstrap required, opened a new log
I20260924 03:12:20.659283 13924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182bef087f54bcdbaeb47017d0283b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } } peers { permanent_uuid: "841eb317eab2452989294b18ebc84950" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } } peers { permanent_uuid: "5f7b370b8bf54a33992fa9c3076db576" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } }
I20260924 03:12:20.659621 13924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:20.659763 13924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7182bef087f54bcdbaeb47017d0283b6, State: Initialized, Role: FOLLOWER
I20260924 03:12:20.660413 13924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182bef087f54bcdbaeb47017d0283b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } } peers { permanent_uuid: "841eb317eab2452989294b18ebc84950" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } } peers { permanent_uuid: "5f7b370b8bf54a33992fa9c3076db576" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } }
I20260924 03:12:20.661537 14099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182bef087f54bcdbaeb47017d0283b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } } peers { permanent_uuid: "841eb317eab2452989294b18ebc84950" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } } peers { permanent_uuid: "5f7b370b8bf54a33992fa9c3076db576" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } } }
I20260924 03:12:20.661908 14099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:12:20.662918 13924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 03:12:20.674974 14110 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:12:20.675117 14110 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:12:20.674013 13857 master_runner.cc:186] Error getting master registration for 127.0.97.61:38257: OK, instance_id { permanent_uuid: "841eb317eab2452989294b18ebc84950" instance_seqno: 1790219539977750 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:12:20.683374 13857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:12:20.709034 13998 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } attempt: 5
W20260924 03:12:20.715355 13878 tablet.cc:2369] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 03:12:20 dist-test-slave-5l98 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 03:12:20.723784 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.709656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44007 (local address 127.0.0.1:51576)
0924 03:12:20.709902 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:20.709922 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:20.710047 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:20.710403 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:20.710407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:20.710822 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:20.710981 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:20.710990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.712038 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.712046 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:20.713004 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:20.713016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.713744 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.713749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:20.713929 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.714554 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:20.714571 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:20.717156 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:20.721006 (+ 3850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:20.721011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:20.721015 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:20.721523 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:20.721999 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:20.722005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:20.722011 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:20.722171 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:20.722817 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:20.722822 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:20.722990 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 03:12:20.723129 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 03:12:20.723402 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 03:12:20.724757 14075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.709707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.0.97.60:44007)
0924 03:12:20.709903 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:20.709911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:20.709973 (+ 62us) server_negotiation.cc:407] Connection header received
0924 03:12:20.710181 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:20.710186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:20.710254 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:20.710372 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:20.711130 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.711888 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.713154 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.713579 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.713736 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.717382 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:20.717387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:20.717390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:20.717438 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:20.720250 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.721661 (+ 1411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.721666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.721669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.721825 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:20.722296 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:20.722303 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:20.722306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:20.722629 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:20.722758 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:20.724096 (+ 1338us) server_negotiation.cc:299] Negotiation successful
0924 03:12:20.724401 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 03:12:20.728926 13998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950: Bootstrap starting.
I20260924 03:12:20.732035 13998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:20.733147 13998 log.cc:824] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950: Log is configured to *not* fsync() on all Append() calls
W20260924 03:12:20.734722 14026 tablet.cc:2369] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:12:20.736368 13998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950: No bootstrap required, opened a new log
I20260924 03:12:20.745169 13998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182bef087f54bcdbaeb47017d0283b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } } peers { permanent_uuid: "841eb317eab2452989294b18ebc84950" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } } peers { permanent_uuid: "5f7b370b8bf54a33992fa9c3076db576" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } }
I20260924 03:12:20.745517 13998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:20.745639 13998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841eb317eab2452989294b18ebc84950, State: Initialized, Role: FOLLOWER
I20260924 03:12:20.746287 13998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182bef087f54bcdbaeb47017d0283b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } } peers { permanent_uuid: "841eb317eab2452989294b18ebc84950" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } } peers { permanent_uuid: "5f7b370b8bf54a33992fa9c3076db576" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } }
I20260924 03:12:20.747236 14113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182bef087f54bcdbaeb47017d0283b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } } peers { permanent_uuid: "841eb317eab2452989294b18ebc84950" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } } peers { permanent_uuid: "5f7b370b8bf54a33992fa9c3076db576" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } } }
I20260924 03:12:20.747681 14113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:12:20.748452 13998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:20.772308 13931 master_runner.cc:263] No leader master found from master 841eb317eab2452989294b18ebc84950
I20260924 03:12:20.772516 13931 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 03:12:20.784250 14124 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:12:20.784521 14124 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:12:20.929455 14085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:12:20.929813 14085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182bef087f54bcdbaeb47017d0283b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } } peers { permanent_uuid: "841eb317eab2452989294b18ebc84950" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } } peers { permanent_uuid: "5f7b370b8bf54a33992fa9c3076db576" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } }
I20260924 03:12:20.931525 14085 leader_election.cc:288] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7182bef087f54bcdbaeb47017d0283b6 (127.0.97.62:38027), 841eb317eab2452989294b18ebc84950 (127.0.97.61:38257)
I20260924 03:12:20.933548 13899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f7b370b8bf54a33992fa9c3076db576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7182bef087f54bcdbaeb47017d0283b6" is_pre_election: true
I20260924 03:12:20.934213 13899 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f7b370b8bf54a33992fa9c3076db576 in term 0.
I20260924 03:12:20.934993 13973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f7b370b8bf54a33992fa9c3076db576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841eb317eab2452989294b18ebc84950" is_pre_election: true
I20260924 03:12:20.935575 13973 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f7b370b8bf54a33992fa9c3076db576 in term 0.
I20260924 03:12:20.936470 14023 leader_election.cc:302] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f7b370b8bf54a33992fa9c3076db576, 841eb317eab2452989294b18ebc84950; no voters:
I20260924 03:12:20.937525 14085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:12:20.937678 14085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:12:20.937778 14085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:20.941999 14085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182bef087f54bcdbaeb47017d0283b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } } peers { permanent_uuid: "841eb317eab2452989294b18ebc84950" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } } peers { permanent_uuid: "5f7b370b8bf54a33992fa9c3076db576" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } }
I20260924 03:12:20.945375 13899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f7b370b8bf54a33992fa9c3076db576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7182bef087f54bcdbaeb47017d0283b6"
I20260924 03:12:20.945616 13899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:20.950495 13899 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f7b370b8bf54a33992fa9c3076db576 in term 1.
I20260924 03:12:20.952163 14023 leader_election.cc:302] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f7b370b8bf54a33992fa9c3076db576, 7182bef087f54bcdbaeb47017d0283b6; no voters:
I20260924 03:12:20.956723 14085 leader_election.cc:288] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [CANDIDATE]: Term 1 election: Requested vote from peers 7182bef087f54bcdbaeb47017d0283b6 (127.0.97.62:38027), 841eb317eab2452989294b18ebc84950 (127.0.97.61:38257)
I20260924 03:12:20.957520 14085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:20.958093 14085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [term 1 LEADER]: Becoming Leader. State: Replica: 5f7b370b8bf54a33992fa9c3076db576, State: Running, Role: LEADER
I20260924 03:12:20.958943 14085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182bef087f54bcdbaeb47017d0283b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } } peers { permanent_uuid: "841eb317eab2452989294b18ebc84950" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } } peers { permanent_uuid: "5f7b370b8bf54a33992fa9c3076db576" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } }
W20260924 03:12:20.966790 13952 tablet.cc:2369] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:12:20.969627 13973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f7b370b8bf54a33992fa9c3076db576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841eb317eab2452989294b18ebc84950"
I20260924 03:12:20.969882 13973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:20.973632 13973 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f7b370b8bf54a33992fa9c3076db576 in term 1.
I20260924 03:12:20.977370 14126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f7b370b8bf54a33992fa9c3076db576. Latest consensus state: current_term: 1 leader_uuid: "5f7b370b8bf54a33992fa9c3076db576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182bef087f54bcdbaeb47017d0283b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } } peers { permanent_uuid: "841eb317eab2452989294b18ebc84950" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } } peers { permanent_uuid: "5f7b370b8bf54a33992fa9c3076db576" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } } }
I20260924 03:12:20.977958 14126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:20.989508 14128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:20.991930 14128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:21.007827 13973 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950 [term 1 FOLLOWER]: Refusing update from remote peer 5f7b370b8bf54a33992fa9c3076db576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:21.009372 14126 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "841eb317eab2452989294b18ebc84950" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:21.021322 13899 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6 [term 1 FOLLOWER]: Refusing update from remote peer 5f7b370b8bf54a33992fa9c3076db576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:21.022403 14126 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7182bef087f54bcdbaeb47017d0283b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:21.058790 14113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f7b370b8bf54a33992fa9c3076db576. Latest consensus state: current_term: 1 leader_uuid: "5f7b370b8bf54a33992fa9c3076db576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182bef087f54bcdbaeb47017d0283b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } } peers { permanent_uuid: "841eb317eab2452989294b18ebc84950" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } } peers { permanent_uuid: "5f7b370b8bf54a33992fa9c3076db576" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } } }
I20260924 03:12:21.059199 14113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:12:21.060761 14099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f7b370b8bf54a33992fa9c3076db576. Latest consensus state: current_term: 1 leader_uuid: "5f7b370b8bf54a33992fa9c3076db576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182bef087f54bcdbaeb47017d0283b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } } peers { permanent_uuid: "841eb317eab2452989294b18ebc84950" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } } peers { permanent_uuid: "5f7b370b8bf54a33992fa9c3076db576" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } } }
I20260924 03:12:21.061089 14099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:12:21.065181 14129 mvcc.cc:205] Tried to move back new op lower bound from 7332739239955304448 to 7332739239783624704. Current Snapshot: MvccSnapshot[applied={T|T < 7332739239955304448}]
I20260924 03:12:21.074458 14130 mvcc.cc:205] Tried to move back new op lower bound from 7332739239955304448 to 7332739239783624704. Current Snapshot: MvccSnapshot[applied={T|T < 7332739239955304448}]
I20260924 03:12:21.082237 14113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f7b370b8bf54a33992fa9c3076db576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182bef087f54bcdbaeb47017d0283b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } } peers { permanent_uuid: "841eb317eab2452989294b18ebc84950" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } } peers { permanent_uuid: "5f7b370b8bf54a33992fa9c3076db576" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } } }
I20260924 03:12:21.082654 14113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:12:21.083402 14085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f7b370b8bf54a33992fa9c3076db576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182bef087f54bcdbaeb47017d0283b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } } peers { permanent_uuid: "841eb317eab2452989294b18ebc84950" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } } peers { permanent_uuid: "5f7b370b8bf54a33992fa9c3076db576" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } } }
I20260924 03:12:21.083773 14085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:21.084178 14085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f7b370b8bf54a33992fa9c3076db576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182bef087f54bcdbaeb47017d0283b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } } peers { permanent_uuid: "841eb317eab2452989294b18ebc84950" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } } peers { permanent_uuid: "5f7b370b8bf54a33992fa9c3076db576" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } } }
I20260924 03:12:21.084551 14085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:21.083026 14131 mvcc.cc:205] Tried to move back new op lower bound from 7332739239955304448 to 7332739239783624704. Current Snapshot: MvccSnapshot[applied={T|T < 7332739239955304448}]
I20260924 03:12:21.088240 14128 catalog_manager.cc:1347] Generated new cluster ID: 8b1cfd184f5047038c4a5c97439feeb3
I20260924 03:12:21.088343 14128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:21.095294 14099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f7b370b8bf54a33992fa9c3076db576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182bef087f54bcdbaeb47017d0283b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38027 } } peers { permanent_uuid: "841eb317eab2452989294b18ebc84950" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38257 } } peers { permanent_uuid: "5f7b370b8bf54a33992fa9c3076db576" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44007 } } }
I20260924 03:12:21.095552 14099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:12:21.158083 14128 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:21.160048 14128 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:21.193477 14128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f7b370b8bf54a33992fa9c3076db576: Generated new TSK 0
I20260924 03:12:21.194607 14128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:21.677304 14110 catalog_manager.cc:1259] Loaded cluster ID: 8b1cfd184f5047038c4a5c97439feeb3
I20260924 03:12:21.677413 14110 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6: loading cluster ID for follower catalog manager: success
I20260924 03:12:21.679872 14110 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6: acquiring CA information for follower catalog manager: success
I20260924 03:12:21.681901 14110 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7182bef087f54bcdbaeb47017d0283b6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:12:21.786933 14124 catalog_manager.cc:1259] Loaded cluster ID: 8b1cfd184f5047038c4a5c97439feeb3
I20260924 03:12:21.787070 14124 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950: loading cluster ID for follower catalog manager: success
I20260924 03:12:21.790082 14124 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950: acquiring CA information for follower catalog manager: success
I20260924 03:12:21.792188 14124 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 841eb317eab2452989294b18ebc84950: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:12:22.297308 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 13857
I20260924 03:12:22.298647 13919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:22.411967 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 13857
I20260924 03:12:22.433122 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 13931
I20260924 03:12:22.434321 13993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:22.544708 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 13931
I20260924 03:12:22.565006 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 14005
I20260924 03:12:22.566334 14067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:22.676149 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 14005
2026-09-24T03:12:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5044 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:22 dist-test-slave-5l98 krb5kdc[14146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:22 dist-test-slave-5l98 krb5kdc[14146](info): set up 2 sockets
Sep 24 03:12:22 dist-test-slave-5l98 krb5kdc[14146](info): commencing operation
krb5kdc: starting...
W20260924 03:12:23.809118 388 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:12:23 dist-test-slave-5l98 krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:12:24.066030 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33677
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:24.406616 14159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:24.406924 14159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:24.407107 14159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:24.414413 14159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:24.414533 14159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:24.414633 14159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:24.414749 14159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:24.414839 14159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:24.424901 14159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:24.426302 14159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:24.427649 14159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:24.440411 14164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:24.440565 14165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:24.440672 14167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:24.441756 14159 server_base.cc:1034] running on GCE node
I20260924 03:12:24.442718 14159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:24.444305 14159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:24.445551 14159 hybrid_clock.cc:648] HybridClock initialized: now 1790219544445484 us; error 54 us; skew 500 ppm
I20260924 03:12:24.445981 14159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:24 dist-test-slave-5l98 krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:24.449388 14159 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:12:24.451167 14159 webserver.cc:460] Webserver started at http://127.0.97.62:36017/ using document root <none> and password file <none>
I20260924 03:12:24.451922 14159 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:24.452098 14159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:24.452433 14159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:24.455615 14159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "d46165f16a8647eeba1d3aeba1e0b747"
format_stamp: "Formatted at 2026-09-24 03:12:24 on dist-test-slave-5l98"
server_key: "c85566f8cb9459829cd114b2f2cecfeb"
server_key_iv: "0220339fb674026dce474ba85d70d1cb"
server_key_version: "encryptionkey@0"
I20260924 03:12:24.456478 14159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "d46165f16a8647eeba1d3aeba1e0b747"
format_stamp: "Formatted at 2026-09-24 03:12:24 on dist-test-slave-5l98"
server_key: "c85566f8cb9459829cd114b2f2cecfeb"
server_key_iv: "0220339fb674026dce474ba85d70d1cb"
server_key_version: "encryptionkey@0"
I20260924 03:12:24.462244 14159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:12:24.466140 14174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:24.467569 14159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:12:24.467787 14159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "d46165f16a8647eeba1d3aeba1e0b747"
format_stamp: "Formatted at 2026-09-24 03:12:24 on dist-test-slave-5l98"
server_key: "c85566f8cb9459829cd114b2f2cecfeb"
server_key_iv: "0220339fb674026dce474ba85d70d1cb"
server_key_version: "encryptionkey@0"
I20260924 03:12:24.468072 14159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:24.486357 14159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:24.491080 14159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:24.491389 14159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:24.510773 14159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33677
I20260924 03:12:24.510803 14225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33677 every 8 connection(s)
I20260924 03:12:24.512216 14159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:24.516310 14226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:24.520005 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 14159
I20260924 03:12:24.520251 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:12:24.521073 388 external_mini_cluster.cc:1428] Setting key e27f4cd2e1be73a8b6fb3e98d8e4e5c1
I20260924 03:12:24.527504 14226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747: Bootstrap starting.
I20260924 03:12:24.531164 14226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:24.532441 14226 log.cc:824] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:24.536194 14226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747: No bootstrap required, opened a new log
Sep 24 03:12:24 dist-test-slave-5l98 krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:12:24.547116 14226 raft_consensus.cc:348] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46165f16a8647eeba1d3aeba1e0b747" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33677 } }
I20260924 03:12:24.547641 14226 raft_consensus.cc:374] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:24.547750 14226 raft_consensus.cc:729] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d46165f16a8647eeba1d3aeba1e0b747, State: Initialized, Role: FOLLOWER
I20260924 03:12:24.548149 14229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:24.522887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41184 (local address 127.0.97.62:33677)
0924 03:12:24.523638 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:24.523649 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:24.523690 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:12:24.526049 (+ 2359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:24.526075 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:24.526473 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:24.526804 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:24.528792 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:24.530248 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:24.532181 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:24.533288 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:24.533519 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:24.538607 (+ 5088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:24.538623 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:24.538642 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:24.538693 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:24.542918 (+ 4225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:24.543775 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:24.543785 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:24.543798 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:24.544053 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:24.544458 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:24.544463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:24.544467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:24.545279 (+ 812us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:24.545607 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:24.546210 (+ 603us) server_negotiation.cc:299] Negotiation successful
0924 03:12:24.546673 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":360,"threads_started":1}
I20260924 03:12:24.548452 14226 consensus_queue.cc:260] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46165f16a8647eeba1d3aeba1e0b747" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33677 } }
I20260924 03:12:24.548776 14226 raft_consensus.cc:388] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:24.548990 14226 raft_consensus.cc:482] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:24.549188 14226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:24.551783 14226 raft_consensus.cc:504] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46165f16a8647eeba1d3aeba1e0b747" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33677 } }
I20260924 03:12:24.552419 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:24.552350 14226 leader_election.cc:302] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d46165f16a8647eeba1d3aeba1e0b747; no voters:
I20260924 03:12:24.552994 14226 leader_election.cc:288] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:24.553112 14231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:24.553824 14231 raft_consensus.cc:686] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747 [term 1 LEADER]: Becoming Leader. State: Replica: d46165f16a8647eeba1d3aeba1e0b747, State: Running, Role: LEADER
I20260924 03:12:24.554450 14231 consensus_queue.cc:237] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46165f16a8647eeba1d3aeba1e0b747" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33677 } }
I20260924 03:12:24.556180 14231 sys_catalog.cc:455] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d46165f16a8647eeba1d3aeba1e0b747. Latest consensus state: current_term: 1 leader_uuid: "d46165f16a8647eeba1d3aeba1e0b747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46165f16a8647eeba1d3aeba1e0b747" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33677 } } }
I20260924 03:12:24.556596 14231 sys_catalog.cc:458] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:24.557070 14233 sys_catalog.cc:455] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d46165f16a8647eeba1d3aeba1e0b747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46165f16a8647eeba1d3aeba1e0b747" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33677 } } }
I20260924 03:12:24.557475 14233 sys_catalog.cc:458] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:24.558380 14226 sys_catalog.cc:564] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:24.558715 14235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:24.565958 14235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:24.576956 14235 catalog_manager.cc:1347] Generated new cluster ID: bb1c8b65527b402d9dee3ae9cde6b290
I20260924 03:12:24.577128 14235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:24.586293 14235 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:24.588148 14235 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:24.601243 14235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d46165f16a8647eeba1d3aeba1e0b747: Generated new TSK 0
I20260924 03:12:24.602275 14235 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:24.616084 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 14159
I20260924 03:12:24.617323 14221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:24.711086 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 14159
2026-09-24T03:12:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2029 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:24 dist-test-slave-5l98 krb5kdc[14254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:24 dist-test-slave-5l98 krb5kdc[14254](info): set up 2 sockets
Sep 24 03:12:24 dist-test-slave-5l98 krb5kdc[14254](info): commencing operation
krb5kdc: starting...
W20260924 03:12:25.861557 388 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:12:26 dist-test-slave-5l98 krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:12:26.113538 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41411
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:26.453366 14267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:26.453732 14267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:26.453927 14267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:26.461360 14267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:26.461472 14267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:26.461575 14267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:26.461679 14267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:26.461764 14267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:26.471676 14267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:26.473129 14267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:26.474512 14267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:26.487071 14273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:26.487102 14272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:26.487033 14275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:26.487901 14267 server_base.cc:1034] running on GCE node
I20260924 03:12:26.488782 14267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:26.490319 14267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:26.491600 14267 hybrid_clock.cc:648] HybridClock initialized: now 1790219546491516 us; error 49 us; skew 500 ppm
I20260924 03:12:26.491963 14267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:26 dist-test-slave-5l98 krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:26.495129 14267 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:12:26.496757 14267 webserver.cc:460] Webserver started at http://127.0.97.62:42705/ using document root <none> and password file <none>
I20260924 03:12:26.497390 14267 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:26.497491 14267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:26.497735 14267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:26.500746 14267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "fb2f76d2322f495bb8b176130b72923c"
format_stamp: "Formatted at 2026-09-24 03:12:26 on dist-test-slave-5l98"
server_key: "c1e7ca83d70edb168a8f8f1618288bb9"
server_key_iv: "fe0301ee266894cfa84c792cb107235c"
server_key_version: "encryptionkey@0"
I20260924 03:12:26.501451 14267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "fb2f76d2322f495bb8b176130b72923c"
format_stamp: "Formatted at 2026-09-24 03:12:26 on dist-test-slave-5l98"
server_key: "c1e7ca83d70edb168a8f8f1618288bb9"
server_key_iv: "fe0301ee266894cfa84c792cb107235c"
server_key_version: "encryptionkey@0"
I20260924 03:12:26.507194 14267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 03:12:26.510932 14282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:26.512477 14267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 03:12:26.512729 14267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "fb2f76d2322f495bb8b176130b72923c"
format_stamp: "Formatted at 2026-09-24 03:12:26 on dist-test-slave-5l98"
server_key: "c1e7ca83d70edb168a8f8f1618288bb9"
server_key_iv: "fe0301ee266894cfa84c792cb107235c"
server_key_version: "encryptionkey@0"
I20260924 03:12:26.512915 14267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:26.535701 14267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:26.539949 14267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:26.540205 14267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:26.558699 14267 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41411
I20260924 03:12:26.558723 14333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41411 every 8 connection(s)
I20260924 03:12:26.560109 14267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:26.564056 14334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:26.567250 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 14267
I20260924 03:12:26.567477 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:12:26.568276 388 external_mini_cluster.cc:1428] Setting key ebcde0a9fd24f13ca0a5a53c3202a193
I20260924 03:12:26.578752 14334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c: Bootstrap starting.
I20260924 03:12:26.582823 14334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:26.584235 14334 log.cc:824] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:12:26 dist-test-slave-5l98 krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:12:26.587361 14334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c: No bootstrap required, opened a new log
I20260924 03:12:26.594020 14337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:26.570716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36846 (local address 127.0.97.62:41411)
0924 03:12:26.571800 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:26.571816 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:26.572058 (+ 242us) server_negotiation.cc:407] Connection header received
0924 03:12:26.574081 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:26.574104 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:26.574515 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:26.574860 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:26.577015 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:26.578443 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:26.580314 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:26.581159 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:26.581303 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:26.586380 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:26.586388 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:26.586404 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:26.586517 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:26.589834 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:26.590592 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:26.590607 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:26.590614 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:26.590777 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:26.591186 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:26.591191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:26.591196 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:26.591731 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:26.592064 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:26.592551 (+ 487us) server_negotiation.cc:299] Negotiation successful
0924 03:12:26.593027 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":447,"threads_started":1}
I20260924 03:12:26.595566 14334 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2f76d2322f495bb8b176130b72923c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } }
I20260924 03:12:26.595920 14334 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:26.596021 14334 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb2f76d2322f495bb8b176130b72923c, State: Initialized, Role: FOLLOWER
I20260924 03:12:26.596669 14334 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2f76d2322f495bb8b176130b72923c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } }
I20260924 03:12:26.596937 14334 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:26.597083 14334 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:26.597250 14334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:26.599328 14334 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2f76d2322f495bb8b176130b72923c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } }
I20260924 03:12:26.599874 14334 leader_election.cc:302] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb2f76d2322f495bb8b176130b72923c; no voters:
I20260924 03:12:26.600580 14334 leader_election.cc:288] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:26.600725 14339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:26.601528 14339 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 1 LEADER]: Becoming Leader. State: Replica: fb2f76d2322f495bb8b176130b72923c, State: Running, Role: LEADER
I20260924 03:12:26.602154 14339 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2f76d2322f495bb8b176130b72923c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } }
I20260924 03:12:26.602790 14334 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:26.605283 14341 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb2f76d2322f495bb8b176130b72923c. Latest consensus state: current_term: 1 leader_uuid: "fb2f76d2322f495bb8b176130b72923c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2f76d2322f495bb8b176130b72923c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } } }
I20260924 03:12:26.605715 14341 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:26.605600 14340 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb2f76d2322f495bb8b176130b72923c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2f76d2322f495bb8b176130b72923c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } } }
I20260924 03:12:26.605989 14340 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:26.609409 14347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:26.615785 14347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:26.625736 14347 catalog_manager.cc:1347] Generated new cluster ID: 3d744971c4ca4c908e14f06ecdb9513f
I20260924 03:12:26.625905 14347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:26.637962 14347 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:26.639875 14347 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:26.653278 14347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c: Generated new TSK 0
I20260924 03:12:26.654398 14347 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 03:12:26.721354 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41411
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:27.071735 14360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:27.072031 14360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:27.072189 14360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:27.079524 14360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:27.079648 14360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:27.079833 14360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:12:27.090083 14360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:27.091396 14360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:27.092918 14360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:27.108182 14365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:27.108203 14366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:27.109045 14368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:27.109083 14360 server_base.cc:1034] running on GCE node
I20260924 03:12:27.109867 14360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:27.110970 14360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:27.112219 14360 hybrid_clock.cc:648] HybridClock initialized: now 1790219547112180 us; error 42 us; skew 500 ppm
I20260924 03:12:27.112665 14360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:27 dist-test-slave-5l98 krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:27.116217 14360 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:12:27.117977 14360 webserver.cc:460] Webserver started at http://127.0.97.1:43533/ using document root <none> and password file <none>
I20260924 03:12:27.118709 14360 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:27.118856 14360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:27.119262 14360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:27.122581 14360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "2a953cfe71cb49dda86b666b75961cc2"
format_stamp: "Formatted at 2026-09-24 03:12:27 on dist-test-slave-5l98"
server_key: "03a7a4305fa28a3175d0b82caf568269"
server_key_iv: "241282bf6e12cfc8fac99b9631cd918f"
server_key_version: "encryptionkey@0"
I20260924 03:12:27.123390 14360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2a953cfe71cb49dda86b666b75961cc2"
format_stamp: "Formatted at 2026-09-24 03:12:27 on dist-test-slave-5l98"
server_key: "03a7a4305fa28a3175d0b82caf568269"
server_key_iv: "241282bf6e12cfc8fac99b9631cd918f"
server_key_version: "encryptionkey@0"
I20260924 03:12:27.128750 14360 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 03:12:27.132277 14375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:27.134009 14360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:12:27.134181 14360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "2a953cfe71cb49dda86b666b75961cc2"
format_stamp: "Formatted at 2026-09-24 03:12:27 on dist-test-slave-5l98"
server_key: "03a7a4305fa28a3175d0b82caf568269"
server_key_iv: "241282bf6e12cfc8fac99b9631cd918f"
server_key_version: "encryptionkey@0"
I20260924 03:12:27.134382 14360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:27.161124 14360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:27.166062 14360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:27.166373 14360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:27.167418 14360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:27.168866 14360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:27.168982 14360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:27.169129 14360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:27.169201 14360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:27.206717 14360 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45013
I20260924 03:12:27.206765 14487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45013 every 8 connection(s)
I20260924 03:12:27.208122 14360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:12:27.218696 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 14360
I20260924 03:12:27.218938 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:12:27.219686 388 external_mini_cluster.cc:1428] Setting key 298d8e1a7588a01b5ffa9206857ca843
Sep 24 03:12:27 dist-test-slave-5l98 krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:12:27.230880 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:27.210990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43053 (local address 127.0.97.62:41411)
0924 03:12:27.211473 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:27.211478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:27.212467 (+ 989us) server_negotiation.cc:407] Connection header received
0924 03:12:27.214401 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:27.214407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:27.214513 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:27.214639 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:27.216115 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:27.217002 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:27.219277 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:27.219809 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:27.219998 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:27.224325 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:27.224330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:27.224334 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:27.224371 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:27.227536 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:27.228378 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:27.228386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:27.228392 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:27.228587 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:27.229038 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:27.229046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:27.229049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:27.229345 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:27.229504 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:27.230257 (+ 753us) server_negotiation.cc:299] Negotiation successful
0924 03:12:27.230509 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":236,"threads_started":1}
I20260924 03:12:27.231916 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:27.211577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:43053)
0924 03:12:27.212310 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:27.212347 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:27.214126 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:27.214864 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:27.214873 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:27.215382 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:27.215943 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:27.215959 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:27.217172 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:27.217177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:27.219057 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:27.219082 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:27.220106 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:27.220117 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:27.220383 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:27.221250 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:27.221276 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:27.224065 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:27.227723 (+ 3658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:27.227734 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:27.227755 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:27.228229 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:27.228704 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:27.228709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:27.228712 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:27.228918 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:27.229491 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:27.229500 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:27.229858 (+ 358us) client_negotiation.cc:769] Sending connection context
0924 03:12:27.230149 (+ 291us) client_negotiation.cc:240] Negotiation successful
0924 03:12:27.230713 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":319,"threads_started":1}
I20260924 03:12:27.233601 14488 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41411
I20260924 03:12:27.233960 14488 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:27.234826 14488 heartbeater.cc:511] Master 127.0.97.62:41411 requested a full tablet report, sending...
I20260924 03:12:27.237251 14299 ts_manager.cc:194] Registered new tserver with Master: 2a953cfe71cb49dda86b666b75961cc2 (127.0.97.1:45013)
I20260924 03:12:27.239023 14299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43053
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 03:12:27.277969 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41411
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:27.617077 14494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:27.617349 14494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:27.617449 14494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:27.624815 14494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:27.624936 14494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:27.625051 14494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:12:27.635202 14494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:27.636413 14494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:27.637724 14494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:27.651269 14499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:27.652661 14494 server_base.cc:1034] running on GCE node
W20260924 03:12:27.655813 14502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:27.656131 14500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:27.656891 14494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:27.657912 14494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:27.659157 14494 hybrid_clock.cc:648] HybridClock initialized: now 1790219547659095 us; error 66 us; skew 500 ppm
I20260924 03:12:27.659495 14494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:27 dist-test-slave-5l98 krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:27.663053 14494 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:12:27.664619 14494 webserver.cc:460] Webserver started at http://127.0.97.2:36163/ using document root <none> and password file <none>
I20260924 03:12:27.665679 14494 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:27.665851 14494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:27.666193 14494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:27.669291 14494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "42041e2066ab4a6d878e1e2a374d5301"
format_stamp: "Formatted at 2026-09-24 03:12:27 on dist-test-slave-5l98"
server_key: "cb8467d9ff0d36dc898c645a0bd6c88c"
server_key_iv: "5669df9398b1f71059ac9823ad3e7f03"
server_key_version: "encryptionkey@0"
I20260924 03:12:27.670100 14494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "42041e2066ab4a6d878e1e2a374d5301"
format_stamp: "Formatted at 2026-09-24 03:12:27 on dist-test-slave-5l98"
server_key: "cb8467d9ff0d36dc898c645a0bd6c88c"
server_key_iv: "5669df9398b1f71059ac9823ad3e7f03"
server_key_version: "encryptionkey@0"
I20260924 03:12:27.675453 14494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:12:27.679153 14509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:27.680590 14494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:12:27.680760 14494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "42041e2066ab4a6d878e1e2a374d5301"
format_stamp: "Formatted at 2026-09-24 03:12:27 on dist-test-slave-5l98"
server_key: "cb8467d9ff0d36dc898c645a0bd6c88c"
server_key_iv: "5669df9398b1f71059ac9823ad3e7f03"
server_key_version: "encryptionkey@0"
I20260924 03:12:27.680970 14494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:27.698681 14494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:27.703466 14494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:27.703799 14494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:27.704919 14494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:27.706501 14494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:27.706614 14494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:27.706770 14494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:27.706840 14494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:27.742902 14494 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33695
I20260924 03:12:27.742952 14621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33695 every 8 connection(s)
I20260924 03:12:27.744202 14494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:12:27.752556 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 14494
I20260924 03:12:27.752768 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:12:27.753206 388 external_mini_cluster.cc:1428] Setting key e1ae4df3d5271cf6a3a64e7021fce2a6
Sep 24 03:12:27 dist-test-slave-5l98 krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:12:27.765440 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:27.746921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49049 (local address 127.0.97.62:41411)
0924 03:12:27.747398 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:27.747403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:27.748299 (+ 896us) server_negotiation.cc:407] Connection header received
0924 03:12:27.750242 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:27.750248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:27.750338 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:27.750467 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:27.751911 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:27.752741 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:27.754461 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:27.754905 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:27.755055 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:27.759353 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:27.759358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:27.759362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:27.759397 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:27.762255 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:27.763016 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:27.763024 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:27.763027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:27.763160 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:27.763611 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:27.763617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:27.763620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:27.763914 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:27.764077 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:27.764824 (+ 747us) server_negotiation.cc:299] Negotiation successful
0924 03:12:27.765076 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":249,"threads_started":1}
I20260924 03:12:27.766445 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:27.747350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.2:49049)
0924 03:12:27.748128 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:27.748169 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:27.749974 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:27.750689 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:27.750697 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:27.751198 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:27.751729 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:27.751744 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:27.752904 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:27.752909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:27.754314 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:27.754327 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:27.755135 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:27.755140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:27.755527 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:27.756287 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:27.756325 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:27.759099 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:27.762413 (+ 3314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:27.762423 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:27.762442 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:27.762876 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:27.763276 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:27.763281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:27.763285 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:27.763489 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:27.764046 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:27.764055 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:27.764401 (+ 346us) client_negotiation.cc:769] Sending connection context
0924 03:12:27.764744 (+ 343us) client_negotiation.cc:240] Negotiation successful
0924 03:12:27.765298 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":341,"threads_started":1}
I20260924 03:12:27.768103 14622 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41411
I20260924 03:12:27.768427 14622 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:27.769274 14622 heartbeater.cc:511] Master 127.0.97.62:41411 requested a full tablet report, sending...
I20260924 03:12:27.770998 14299 ts_manager.cc:194] Registered new tserver with Master: 42041e2066ab4a6d878e1e2a374d5301 (127.0.97.2:33695)
I20260924 03:12:27.772286 14299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49049
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 03:12:27.808665 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41411
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:28.170660 14628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:28.170977 14628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:28.171082 14628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:28.178676 14628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:28.178797 14628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:28.178910 14628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:12:28.189183 14628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:28.190516 14628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:28.191946 14628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:28.209724 14633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:28.209822 14634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:28.209883 14636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:28.214423 14628 server_base.cc:1034] running on GCE node
I20260924 03:12:28.215206 14628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:28.216279 14628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:28.217585 14628 hybrid_clock.cc:648] HybridClock initialized: now 1790219548217550 us; error 71 us; skew 500 ppm
I20260924 03:12:28.217999 14628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:28 dist-test-slave-5l98 krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:28.226389 14628 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:12:28.228129 14628 webserver.cc:460] Webserver started at http://127.0.97.3:35959/ using document root <none> and password file <none>
I20260924 03:12:28.228953 14628 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:28.229094 14628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:28.229440 14628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:28.232393 14628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "78f63d155fdc4959869cf5134eb15498"
format_stamp: "Formatted at 2026-09-24 03:12:28 on dist-test-slave-5l98"
server_key: "32c4add8bf937c5ba7b2250f553dcd53"
server_key_iv: "ab99e99ab4673ec8f5b2a1c9b42bb492"
server_key_version: "encryptionkey@0"
I20260924 03:12:28.233266 14628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "78f63d155fdc4959869cf5134eb15498"
format_stamp: "Formatted at 2026-09-24 03:12:28 on dist-test-slave-5l98"
server_key: "32c4add8bf937c5ba7b2250f553dcd53"
server_key_iv: "ab99e99ab4673ec8f5b2a1c9b42bb492"
server_key_version: "encryptionkey@0"
I20260924 03:12:28.239004 14628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 03:12:28.242767 14643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:28.242834 14488 heartbeater.cc:503] Master 127.0.97.62:41411 was elected leader, sending a full tablet report...
I20260924 03:12:28.244421 14628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 03:12:28.244709 14628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "78f63d155fdc4959869cf5134eb15498"
format_stamp: "Formatted at 2026-09-24 03:12:28 on dist-test-slave-5l98"
server_key: "32c4add8bf937c5ba7b2250f553dcd53"
server_key_iv: "ab99e99ab4673ec8f5b2a1c9b42bb492"
server_key_version: "encryptionkey@0"
I20260924 03:12:28.244982 14628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:28.260197 14628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:28.264956 14628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:28.265321 14628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:28.266321 14628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:28.267714 14628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:28.267791 14628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:28.267946 14628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:28.268019 14628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:28.306277 14628 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38731
I20260924 03:12:28.306360 14755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38731 every 8 connection(s)
I20260924 03:12:28.307626 14628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:12:28.316705 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 14628
I20260924 03:12:28.316891 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:12:28.317481 388 external_mini_cluster.cc:1428] Setting key 18ee87f295b956718d980f257f17e779
Sep 24 03:12:28 dist-test-slave-5l98 krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:12:28.330296 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:28.310394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49863 (local address 127.0.97.62:41411)
0924 03:12:28.310927 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:28.310934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:28.311876 (+ 942us) server_negotiation.cc:407] Connection header received
0924 03:12:28.313937 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:28.313947 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:28.314020 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:28.314119 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:28.315666 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.316866 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.319148 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.319656 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.319803 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:28.323845 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:28.323851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:28.323855 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:28.323894 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:28.326911 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:28.327850 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:28.327854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:28.327857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:28.327992 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:28.328413 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:28.328417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:28.328419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:28.328858 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:28.328986 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:28.329788 (+ 802us) server_negotiation.cc:299] Negotiation successful
0924 03:12:28.330006 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":257,"threads_started":1}
I20260924 03:12:28.331226 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:28.310941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.3:49863)
0924 03:12:28.311697 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:28.311737 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:28.313657 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:28.314352 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:28.314361 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:28.314895 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:28.315484 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:28.315501 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.317093 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.317100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:28.319008 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:28.319022 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.319887 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.319897 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:28.320173 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:28.320950 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:28.320976 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:28.323588 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:28.327077 (+ 3489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:28.327085 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:28.327099 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:28.327709 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:28.328119 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:28.328123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:28.328125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:28.328308 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:28.329037 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:28.329043 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:28.329398 (+ 355us) client_negotiation.cc:769] Sending connection context
0924 03:12:28.329653 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 03:12:28.330062 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":276,"threads_started":1}
I20260924 03:12:28.332890 14756 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41411
I20260924 03:12:28.333215 14756 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:28.334007 14756 heartbeater.cc:511] Master 127.0.97.62:41411 requested a full tablet report, sending...
I20260924 03:12:28.335379 14298 ts_manager.cc:194] Registered new tserver with Master: 78f63d155fdc4959869cf5134eb15498 (127.0.97.3:38731)
I20260924 03:12:28.336215 14298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49863
I20260924 03:12:28.347268 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:12:28.367767 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:28.352902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36858 (local address 127.0.97.62:41411)
0924 03:12:28.353107 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:28.353111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:28.353574 (+ 463us) server_negotiation.cc:407] Connection header received
0924 03:12:28.353701 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:28.353705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:28.353790 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:28.353896 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:28.355394 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.356309 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.358036 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.358536 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.358672 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:28.361561 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:28.361566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:28.361570 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:28.361610 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:28.364624 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:28.365515 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:28.365520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:28.365522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:28.365671 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:28.366191 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:28.366195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:28.366197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:28.366642 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:28.366860 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:28.367258 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 03:12:28.367457 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 03:12:28.373263 14298 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:12:28.380370 14298 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:12:28.411947 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:28.394255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.0.97.1:45013)
0924 03:12:28.402882 (+ 8627us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:28.402891 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:28.402914 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:12:28.403049 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:28.403055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:28.403273 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:28.404634 (+ 1361us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:28.404928 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.406388 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.409491 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.410714 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.410871 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:28.411122 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:28.411453 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 03:12:28.411622 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8459,"thread_start_us":8290,"threads_started":1}
I20260924 03:12:28.411996 14770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:28.393954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45013 (local address 127.0.0.1:42698)
0924 03:12:28.401082 (+ 7128us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:28.401134 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:28.401355 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:28.403538 (+ 2183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:28.403548 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:28.403580 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:28.404283 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:28.404302 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.406523 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.406531 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:28.409264 (+ 2733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:28.409279 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.410933 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.410948 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:28.411187 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:28.411241 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 03:12:28.411390 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 03:12:28.411603 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6854,"thread_start_us":8150,"threads_started":1}
I20260924 03:12:28.413363 14772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:28.394675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33696 (local address 127.0.97.2:33695)
0924 03:12:28.402004 (+ 7329us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:28.402016 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:28.402840 (+ 824us) server_negotiation.cc:407] Connection header received
0924 03:12:28.402986 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:28.402996 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:28.403275 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:28.403476 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:28.405475 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.407172 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.409723 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.411424 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.411552 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:28.411754 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:28.412787 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0924 03:12:28.412992 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7167,"thread_start_us":7046,"threads_started":1}
I20260924 03:12:28.414237 14769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:28.394146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33695 (local address 127.0.0.1:33696)
0924 03:12:28.402376 (+ 8230us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:28.402398 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:28.402534 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:28.404109 (+ 1575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:28.404117 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:28.404162 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:28.405259 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:28.405281 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.407322 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.407327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:28.409567 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:28.409584 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.412296 (+ 2712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.412308 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:28.412541 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:28.412587 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 03:12:28.413590 (+ 1003us) client_negotiation.cc:240] Negotiation successful
0924 03:12:28.413751 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8105,"mutex_wait_us":32,"thread_start_us":6508,"threads_started":1}
I20260924 03:12:28.414872 14773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:28.402811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38731 (local address 127.0.0.1:50176)
0924 03:12:28.405646 (+ 2835us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:28.405665 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:28.405842 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:28.406678 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:28.406688 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:28.406715 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:28.407127 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:28.407139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.409362 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.409370 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:28.411792 (+ 2422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:28.411807 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.413978 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.413985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:28.414166 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:28.414197 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 03:12:28.414306 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 03:12:28.414589 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2205,"thread_start_us":1838,"threads_started":1}
I20260924 03:12:28.415138 14774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:28.402813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50176 (local address 127.0.97.3:38731)
0924 03:12:28.404331 (+ 1518us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:28.404343 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:28.405750 (+ 1407us) server_negotiation.cc:407] Connection header received
0924 03:12:28.406192 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:28.406202 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:28.406452 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:28.406679 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:28.407322 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.409123 (+ 1801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.412006 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.413750 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.413910 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:28.414137 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:28.414398 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 03:12:28.414792 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1354,"thread_start_us":1017,"threads_started":1}
I20260924 03:12:28.416644 14423 tablet_service.cc:1467] Processing CreateTablet for tablet 6665fd6bd98c4b44a0f1b49a9e9927e6 (DEFAULT_TABLE table=test-table [id=46cfc68e871241b3a2703d561631b75c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:28.418211 14423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6665fd6bd98c4b44a0f1b49a9e9927e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:28.418283 14557 tablet_service.cc:1467] Processing CreateTablet for tablet 6665fd6bd98c4b44a0f1b49a9e9927e6 (DEFAULT_TABLE table=test-table [id=46cfc68e871241b3a2703d561631b75c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:28.418974 14691 tablet_service.cc:1467] Processing CreateTablet for tablet 6665fd6bd98c4b44a0f1b49a9e9927e6 (DEFAULT_TABLE table=test-table [id=46cfc68e871241b3a2703d561631b75c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:28.419759 14557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6665fd6bd98c4b44a0f1b49a9e9927e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:28.420398 14691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6665fd6bd98c4b44a0f1b49a9e9927e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:28.430794 14775 tablet_bootstrap.cc:492] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2: Bootstrap starting.
I20260924 03:12:28.434159 14775 tablet_bootstrap.cc:654] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:28.435220 14775 log.cc:824] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:28.436268 14776 tablet_bootstrap.cc:492] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301: Bootstrap starting.
I20260924 03:12:28.440538 14775 tablet_bootstrap.cc:492] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2: No bootstrap required, opened a new log
I20260924 03:12:28.438499 14777 tablet_bootstrap.cc:492] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498: Bootstrap starting.
I20260924 03:12:28.440826 14775 ts_tablet_manager.cc:1397] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20260924 03:12:28.441263 14776 tablet_bootstrap.cc:654] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:28.442525 14776 log.cc:824] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:28.444015 14777 tablet_bootstrap.cc:654] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:28.445202 14777 log.cc:824] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:28.446040 14776 tablet_bootstrap.cc:492] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301: No bootstrap required, opened a new log
I20260924 03:12:28.446700 14776 ts_tablet_manager.cc:1397] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 03:12:28.451409 14777 tablet_bootstrap.cc:492] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498: No bootstrap required, opened a new log
I20260924 03:12:28.451701 14777 ts_tablet_manager.cc:1397] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.008s
I20260924 03:12:28.451436 14775 raft_consensus.cc:348] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } } peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } }
I20260924 03:12:28.451934 14775 raft_consensus.cc:374] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:28.452041 14775 raft_consensus.cc:729] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a953cfe71cb49dda86b666b75961cc2, State: Initialized, Role: FOLLOWER
I20260924 03:12:28.452698 14775 consensus_queue.cc:260] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } } peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } }
I20260924 03:12:28.459136 14775 ts_tablet_manager.cc:1428] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2: Time spent starting tablet: real 0.018s user 0.011s sys 0.006s
I20260924 03:12:28.459801 14776 raft_consensus.cc:348] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } } peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } }
I20260924 03:12:28.460316 14776 raft_consensus.cc:374] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:28.460438 14776 raft_consensus.cc:729] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42041e2066ab4a6d878e1e2a374d5301, State: Initialized, Role: FOLLOWER
W20260924 03:12:28.460840 14489 tablet.cc:2369] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:12:28.460948 14777 raft_consensus.cc:348] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } } peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } }
I20260924 03:12:28.461386 14777 raft_consensus.cc:374] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:28.461248 14776 consensus_queue.cc:260] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } } peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } }
I20260924 03:12:28.461483 14777 raft_consensus.cc:729] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f63d155fdc4959869cf5134eb15498, State: Initialized, Role: FOLLOWER
I20260924 03:12:28.462143 14777 consensus_queue.cc:260] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } } peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } }
I20260924 03:12:28.462770 14622 heartbeater.cc:503] Master 127.0.97.62:41411 was elected leader, sending a full tablet report...
I20260924 03:12:28.463364 14776 ts_tablet_manager.cc:1428] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301: Time spent starting tablet: real 0.017s user 0.010s sys 0.004s
I20260924 03:12:28.463842 14756 heartbeater.cc:503] Master 127.0.97.62:41411 was elected leader, sending a full tablet report...
I20260924 03:12:28.464035 14777 ts_tablet_manager.cc:1428] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
W20260924 03:12:28.496404 14623 tablet.cc:2369] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:12:28.533123 14781 raft_consensus.cc:482] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:12:28.533604 14781 raft_consensus.cc:504] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } } peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } }
I20260924 03:12:28.535578 14781 leader_election.cc:288] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42041e2066ab4a6d878e1e2a374d5301 (127.0.97.2:33695), 78f63d155fdc4959869cf5134eb15498 (127.0.97.3:38731)
I20260924 03:12:28.543727 14784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:28.535989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33695 (local address 127.0.97.1:41125)
0924 03:12:28.537055 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:28.537070 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:28.537194 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:28.537608 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:28.537612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:28.537638 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:28.538223 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:28.538235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.539449 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.539454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:28.541212 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:28.541225 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.542762 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.542768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:28.542947 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:28.542985 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 03:12:28.543095 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 03:12:28.543234 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":46,"spinlock_wait_cycles":23424,"thread_start_us":690,"threads_started":1}
I20260924 03:12:28.543718 14772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:28.536140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41125 (local address 127.0.97.2:33695)
0924 03:12:28.536407 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:28.536413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:28.537164 (+ 751us) server_negotiation.cc:407] Connection header received
0924 03:12:28.537362 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:28.537371 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:28.537472 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:28.537576 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:28.538381 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.539318 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.541390 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.542531 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.542644 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:28.542724 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:28.543162 (+ 438us) server_negotiation.cc:299] Negotiation successful
0924 03:12:28.543419 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 03:12:28.544571 14774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:28.536186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33289 (local address 127.0.97.3:38731)
0924 03:12:28.536421 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:28.536430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:28.537478 (+ 1048us) server_negotiation.cc:407] Connection header received
0924 03:12:28.537583 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:28.537588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:28.537683 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:28.537781 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:28.538531 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.539681 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.542037 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.543172 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.543285 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:28.543384 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:28.544064 (+ 680us) server_negotiation.cc:299] Negotiation successful
0924 03:12:28.544236 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 03:12:28.545208 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:28.536117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38731 (local address 127.0.97.1:33289)
0924 03:12:28.537310 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:28.537327 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:28.537456 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:28.537804 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:28.537807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:28.537841 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:28.538339 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:28.538354 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.539833 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.539837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:28.541854 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:28.541870 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.543611 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.543617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:28.543846 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:28.543887 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 03:12:28.544758 (+ 871us) client_negotiation.cc:240] Negotiation successful
0924 03:12:28.544923 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"mutex_wait_us":18,"spinlock_wait_cycles":896,"thread_start_us":654,"threads_started":1}
I20260924 03:12:28.545092 14577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6665fd6bd98c4b44a0f1b49a9e9927e6" candidate_uuid: "2a953cfe71cb49dda86b666b75961cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42041e2066ab4a6d878e1e2a374d5301" is_pre_election: true
I20260924 03:12:28.545701 14577 raft_consensus.cc:2454] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a953cfe71cb49dda86b666b75961cc2 in term 0.
I20260924 03:12:28.546319 14711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6665fd6bd98c4b44a0f1b49a9e9927e6" candidate_uuid: "2a953cfe71cb49dda86b666b75961cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f63d155fdc4959869cf5134eb15498" is_pre_election: true
I20260924 03:12:28.546505 14379 leader_election.cc:302] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a953cfe71cb49dda86b666b75961cc2, 42041e2066ab4a6d878e1e2a374d5301; no voters:
I20260924 03:12:28.546979 14781 raft_consensus.cc:2781] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:12:28.546926 14711 raft_consensus.cc:2454] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a953cfe71cb49dda86b666b75961cc2 in term 0.
I20260924 03:12:28.547147 14781 raft_consensus.cc:482] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:12:28.547276 14781 raft_consensus.cc:3037] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:28.550304 14781 raft_consensus.cc:504] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } } peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } }
I20260924 03:12:28.550982 14781 leader_election.cc:288] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 1 election: Requested vote from peers 42041e2066ab4a6d878e1e2a374d5301 (127.0.97.2:33695), 78f63d155fdc4959869cf5134eb15498 (127.0.97.3:38731)
I20260924 03:12:28.551502 14577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6665fd6bd98c4b44a0f1b49a9e9927e6" candidate_uuid: "2a953cfe71cb49dda86b666b75961cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42041e2066ab4a6d878e1e2a374d5301"
I20260924 03:12:28.551656 14711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6665fd6bd98c4b44a0f1b49a9e9927e6" candidate_uuid: "2a953cfe71cb49dda86b666b75961cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f63d155fdc4959869cf5134eb15498"
I20260924 03:12:28.551712 14577 raft_consensus.cc:3037] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:28.551899 14711 raft_consensus.cc:3037] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:28.554183 14711 raft_consensus.cc:2454] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a953cfe71cb49dda86b666b75961cc2 in term 1.
I20260924 03:12:28.554302 14577 raft_consensus.cc:2454] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a953cfe71cb49dda86b666b75961cc2 in term 1.
I20260924 03:12:28.554687 14377 leader_election.cc:302] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a953cfe71cb49dda86b666b75961cc2, 78f63d155fdc4959869cf5134eb15498; no voters:
I20260924 03:12:28.555016 14781 raft_consensus.cc:2781] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:28.555577 14781 raft_consensus.cc:686] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 [term 1 LEADER]: Becoming Leader. State: Replica: 2a953cfe71cb49dda86b666b75961cc2, State: Running, Role: LEADER
I20260924 03:12:28.556145 14781 consensus_queue.cc:237] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } } peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } }
W20260924 03:12:28.559444 14757 tablet.cc:2369] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:12:28.561631 14299 catalog_manager.cc:5495] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a953cfe71cb49dda86b666b75961cc2 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "2a953cfe71cb49dda86b666b75961cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } health_report { overall_health: UNKNOWN } } }
I20260924 03:12:28.612061 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:28.606178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.0.97.1:45013)
0924 03:12:28.606413 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:28.606419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:28.606504 (+ 85us) server_negotiation.cc:407] Connection header received
0924 03:12:28.606762 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:28.606766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:28.606881 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:28.607039 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:12:28.607421 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.608158 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.609871 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.610281 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.610416 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:28.610938 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:12:28.611438 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:12:28.611594 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:28.611701 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 03:12:28.611784 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 03:12:28.620831 14577 raft_consensus.cc:1261] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301 [term 1 FOLLOWER]: Refusing update from remote peer 2a953cfe71cb49dda86b666b75961cc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:28.621033 14711 raft_consensus.cc:1261] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498 [term 1 FOLLOWER]: Refusing update from remote peer 2a953cfe71cb49dda86b666b75961cc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:28.621944 14786 consensus_queue.cc:1035] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:28.622284 14781 consensus_queue.cc:1035] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:28.649840 14299 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:12:28.650238 14299 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:12:28.653904 14299 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c: Sending DeleteTablet for 3 replicas of tablet 6665fd6bd98c4b44a0f1b49a9e9927e6
I20260924 03:12:28.655157 14557 tablet_service.cc:1514] Processing DeleteTablet for tablet 6665fd6bd98c4b44a0f1b49a9e9927e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33696
I20260924 03:12:28.655371 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 14360
I20260924 03:12:28.655438 14423 tablet_service.cc:1514] Processing DeleteTablet for tablet 6665fd6bd98c4b44a0f1b49a9e9927e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42698
I20260924 03:12:28.656922 14691 tablet_service.cc:1514] Processing DeleteTablet for tablet 6665fd6bd98c4b44a0f1b49a9e9927e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50176
I20260924 03:12:28.679621 14801 tablet_replica.cc:330] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498: stopping tablet replica
I20260924 03:12:28.681326 14801 raft_consensus.cc:2229] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:12:28.684623 14801 raft_consensus.cc:2258] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:28.689548 14801 ts_tablet_manager.cc:1905] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:28.692739 14798 tablet_replica.cc:330] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301: stopping tablet replica
I20260924 03:12:28.693161 14798 raft_consensus.cc:2229] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:12:28.693636 14798 raft_consensus.cc:2258] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:28.694437 14799 tablet_replica.cc:330] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2: stopping tablet replica
I20260924 03:12:28.694795 14799 raft_consensus.cc:2229] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:12:28.695391 14799 raft_consensus.cc:2258] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:28.697723 14799 ts_tablet_manager.cc:1905] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:28.698263 14801 ts_tablet_manager.cc:1918] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:28.698426 14801 log.cc:1196] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/6665fd6bd98c4b44a0f1b49a9e9927e6
I20260924 03:12:28.698943 14801 ts_tablet_manager.cc:1939] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 78f63d155fdc4959869cf5134eb15498: Deleting consensus metadata
I20260924 03:12:28.700892 14284 catalog_manager.cc:4841] TS 78f63d155fdc4959869cf5134eb15498 (127.0.97.3:38731): tablet 6665fd6bd98c4b44a0f1b49a9e9927e6 (table test-table [id=46cfc68e871241b3a2703d561631b75c]) successfully deleted
I20260924 03:12:28.703444 14799 ts_tablet_manager.cc:1918] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:28.703629 14799 log.cc:1196] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/6665fd6bd98c4b44a0f1b49a9e9927e6
I20260924 03:12:28.704211 14799 ts_tablet_manager.cc:1939] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 2a953cfe71cb49dda86b666b75961cc2: Deleting consensus metadata
I20260924 03:12:28.704981 14798 ts_tablet_manager.cc:1905] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:28.705909 14284 catalog_manager.cc:4841] TS 2a953cfe71cb49dda86b666b75961cc2 (127.0.97.1:45013): tablet 6665fd6bd98c4b44a0f1b49a9e9927e6 (table test-table [id=46cfc68e871241b3a2703d561631b75c]) successfully deleted
Sep 24 03:12:28 dist-test-slave-5l98 krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:12:28.711180 14798 ts_tablet_manager.cc:1918] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:28.711369 14798 log.cc:1196] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/6665fd6bd98c4b44a0f1b49a9e9927e6
I20260924 03:12:28.711923 14798 ts_tablet_manager.cc:1939] T 6665fd6bd98c4b44a0f1b49a9e9927e6 P 42041e2066ab4a6d878e1e2a374d5301: Deleting consensus metadata
I20260924 03:12:28.713891 14286 catalog_manager.cc:4841] TS 42041e2066ab4a6d878e1e2a374d5301 (127.0.97.2:33695): tablet 6665fd6bd98c4b44a0f1b49a9e9927e6 (table test-table [id=46cfc68e871241b3a2703d561631b75c]) successfully deleted
I20260924 03:12:28.717773 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:28.657732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42716 (local address 127.0.97.1:45013)
0924 03:12:28.657997 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:28.658006 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:28.680681 (+ 22675us) server_negotiation.cc:407] Connection header received
0924 03:12:28.682028 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:28.682037 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:28.682154 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:28.682273 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:28.687596 (+ 5323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.689361 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.696021 (+ 6660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.696740 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.696850 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:28.710018 (+ 13168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:28.710032 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:28.710050 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:28.710100 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:28.714299 (+ 4199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:28.715256 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:28.715263 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:28.715265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:28.715416 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:28.715851 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:28.715854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:28.715856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:28.716318 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:28.716649 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:28.717070 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 03:12:28.717412 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:12:28.718552 14483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:28.827222 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 14360
I20260924 03:12:28.846097 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 14494
Sep 24 03:12:28 dist-test-slave-5l98 krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 03:12:28.867414 14772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:28.846967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.0.97.2:33695)
0924 03:12:28.847187 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:28.847195 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:28.847243 (+ 48us) server_negotiation.cc:407] Connection header received
0924 03:12:28.847315 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:28.847319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:28.847412 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:28.847548 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:28.849002 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.849855 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.851171 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.851550 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.851684 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:28.860888 (+ 9204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:28.860906 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:28.860930 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:28.860983 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:28.864156 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:28.865098 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:28.865106 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:28.865108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:28.865258 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:28.865679 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:28.865684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:28.865686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:28.866133 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:28.866448 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:28.866814 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 03:12:28.867057 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 03:12:28.868139 14617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:28.962417 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 14494
I20260924 03:12:28.979809 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 14628
Sep 24 03:12:28 dist-test-slave-5l98 krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 03:12:28.996135 14774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:28.980898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50190 (local address 127.0.97.3:38731)
0924 03:12:28.981075 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:28.981079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:28.981097 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:12:28.981163 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:28.981167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:28.981243 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:28.981328 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:28.982535 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.983287 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.984743 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:28.985166 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:28.985286 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:28.990019 (+ 4733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:28.990035 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:28.990051 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:28.990098 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:28.993359 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:28.994024 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:28.994037 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:28.994044 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:28.994198 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:28.994580 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:28.994584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:28.994586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:28.994949 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:28.995165 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:28.995557 (+ 392us) server_negotiation.cc:299] Negotiation successful
0924 03:12:28.995782 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 03:12:28.997027 14751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:29.089936 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 14628
I20260924 03:12:29.107349 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 14267
I20260924 03:12:29.109095 14329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:29.223829 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 14267
I20260924 03:12:29.245182 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41411
--webserver_interface=127.0.97.62
--webserver_port=42705
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:29.583242 14806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:29.583561 14806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:29.583742 14806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:29.590960 14806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:29.591068 14806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:29.591163 14806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:29.591248 14806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:29.591354 14806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:29.601315 14806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:29.602656 14806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:29.603968 14806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:29.627007 14813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:29.627045 14815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:29.627069 14812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:29.627967 14806 server_base.cc:1034] running on GCE node
I20260924 03:12:29.628973 14806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:29.630609 14806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:29.631891 14806 hybrid_clock.cc:648] HybridClock initialized: now 1790219549631837 us; error 56 us; skew 500 ppm
I20260924 03:12:29.632356 14806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:29 dist-test-slave-5l98 krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:29.636334 14806 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:12:29.638032 14806 webserver.cc:460] Webserver started at http://127.0.97.62:42705/ using document root <none> and password file <none>
I20260924 03:12:29.638769 14806 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:29.638887 14806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:29.645119 14806 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260924 03:12:29.648398 14822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:29.649956 14806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:12:29.650111 14806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "fb2f76d2322f495bb8b176130b72923c"
format_stamp: "Formatted at 2026-09-24 03:12:26 on dist-test-slave-5l98"
server_key: "c1e7ca83d70edb168a8f8f1618288bb9"
server_key_iv: "fe0301ee266894cfa84c792cb107235c"
server_key_version: "encryptionkey@0"
I20260924 03:12:29.651240 14806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:29.673383 14806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:29.678020 14806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:29.678269 14806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:29.698393 14806 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41411
I20260924 03:12:29.698414 14873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41411 every 8 connection(s)
I20260924 03:12:29.700026 14806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:29.707104 14874 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:12:29.709100 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 14806
I20260924 03:12:29.709986 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45013
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41411
--webserver_port=43533
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:12:29.710666 14874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c: Bootstrap starting.
I20260924 03:12:29.730778 14874 log.cc:824] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:29.743028 14874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:12:29.743604 14874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c: Bootstrap complete.
I20260924 03:12:29.753233 14874 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2f76d2322f495bb8b176130b72923c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } }
I20260924 03:12:29.754138 14874 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb2f76d2322f495bb8b176130b72923c, State: Initialized, Role: FOLLOWER
I20260924 03:12:29.754760 14874 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2f76d2322f495bb8b176130b72923c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } }
I20260924 03:12:29.755018 14874 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:29.755187 14874 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:29.755357 14874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:12:29.759343 14874 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2f76d2322f495bb8b176130b72923c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } }
I20260924 03:12:29.759873 14874 leader_election.cc:302] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb2f76d2322f495bb8b176130b72923c; no voters:
I20260924 03:12:29.760588 14874 leader_election.cc:288] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 03:12:29.760756 14878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:12:29.761914 14878 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 2 LEADER]: Becoming Leader. State: Replica: fb2f76d2322f495bb8b176130b72923c, State: Running, Role: LEADER
I20260924 03:12:29.762583 14878 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2f76d2322f495bb8b176130b72923c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } }
I20260924 03:12:29.763036 14874 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:29.765344 14879 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fb2f76d2322f495bb8b176130b72923c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2f76d2322f495bb8b176130b72923c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } } }
I20260924 03:12:29.765810 14879 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:29.769665 14880 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb2f76d2322f495bb8b176130b72923c. Latest consensus state: current_term: 2 leader_uuid: "fb2f76d2322f495bb8b176130b72923c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2f76d2322f495bb8b176130b72923c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } } }
I20260924 03:12:29.770099 14880 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:29.771858 14884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:29.783342 14884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:29.785301 14884 catalog_manager.cc:1259] Loaded cluster ID: 3d744971c4ca4c908e14f06ecdb9513f
I20260924 03:12:29.785832 14884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:29.788386 14884 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 03:12:29.799295 14884 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 03:12:29.806392 14884 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:29.807665 14884 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c: Loaded TSK: 0
I20260924 03:12:29.808833 14884 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:12:30.082171 14876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:30.082530 14876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:30.082710 14876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:30.090477 14876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:30.090652 14876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:30.090834 14876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:12:30.101605 14876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43533
--tserver_master_addrs=127.0.97.62:41411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:30.103076 14876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:30.104557 14876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:30.119761 14902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:30.120014 14904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:30.119745 14901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:30.126582 14876 server_base.cc:1034] running on GCE node
I20260924 03:12:30.127465 14876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:30.128942 14876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:30.130263 14876 hybrid_clock.cc:648] HybridClock initialized: now 1790219550130201 us; error 62 us; skew 500 ppm
I20260924 03:12:30.130679 14876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:30 dist-test-slave-5l98 krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:30.135017 14876 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:12:30.136951 14876 webserver.cc:460] Webserver started at http://127.0.97.1:43533/ using document root <none> and password file <none>
I20260924 03:12:30.137748 14876 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:30.137902 14876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:30.144080 14876 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260924 03:12:30.147248 14911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:30.149021 14876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 03:12:30.149209 14876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "2a953cfe71cb49dda86b666b75961cc2"
format_stamp: "Formatted at 2026-09-24 03:12:27 on dist-test-slave-5l98"
server_key: "03a7a4305fa28a3175d0b82caf568269"
server_key_iv: "241282bf6e12cfc8fac99b9631cd918f"
server_key_version: "encryptionkey@0"
I20260924 03:12:30.150349 14876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:30.163380 14876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:30.168341 14876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:30.168718 14876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:30.169996 14876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:30.171571 14876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:30.171715 14876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:30.171859 14876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:30.171988 14876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:30.225171 14876 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45013
I20260924 03:12:30.225230 15023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45013 every 8 connection(s)
I20260924 03:12:30.226907 14876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:12:30.234709 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 14876
I20260924 03:12:30.235344 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33695
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41411
--webserver_port=36163
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:12:30 dist-test-slave-5l98 krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:12:30.251626 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:30.230225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:52165)
0924 03:12:30.231033 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:30.231069 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:30.232838 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:30.234207 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:30.234214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:30.234841 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:30.235447 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:30.235473 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:30.237269 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:30.237273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:30.238513 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:30.238531 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:30.239575 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:30.239583 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:30.239852 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:30.240629 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:30.240653 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:30.243244 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:30.246895 (+ 3651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:30.246905 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:30.246923 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:30.247414 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:30.247881 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:30.247889 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:30.247892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:30.248104 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:30.248952 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:30.248961 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:30.249381 (+ 420us) client_negotiation.cc:769] Sending connection context
0924 03:12:30.249695 (+ 314us) client_negotiation.cc:240] Negotiation successful
0924 03:12:30.250294 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":361,"threads_started":1}
I20260924 03:12:30.251637 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:30.230139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52165 (local address 127.0.97.62:41411)
0924 03:12:30.230893 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:30.230909 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:30.231199 (+ 290us) server_negotiation.cc:407] Connection header received
0924 03:12:30.233399 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:30.233423 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:30.233730 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:30.234047 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:30.235714 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:30.237087 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:30.238700 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:30.239321 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:30.239530 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:30.243563 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:30.243579 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:30.243596 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:30.243642 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:30.246710 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:30.247583 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:30.247591 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:30.247596 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:30.247758 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:30.248226 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:30.248233 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:30.248236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:30.248792 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:30.249029 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:30.249932 (+ 903us) server_negotiation.cc:299] Negotiation successful
0924 03:12:30.250320 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":314,"threads_started":1}
I20260924 03:12:30.254925 15024 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41411
I20260924 03:12:30.255311 15024 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:30.256172 15024 heartbeater.cc:511] Master 127.0.97.62:41411 requested a full tablet report, sending...
I20260924 03:12:30.258674 14839 ts_manager.cc:194] Registered new tserver with Master: 2a953cfe71cb49dda86b666b75961cc2 (127.0.97.1:45013)
I20260924 03:12:30.260826 14839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52165
W20260924 03:12:30.578411 15028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:30.578708 15028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:30.578804 15028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:30.586205 15028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:30.586338 15028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:30.586453 15028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:12:30.596518 15028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36163
--tserver_master_addrs=127.0.97.62:41411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:30.597795 15028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:30.599136 15028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:30.613106 15034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:30.613178 15036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:30.613154 15033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:30.613482 15028 server_base.cc:1034] running on GCE node
I20260924 03:12:30.614359 15028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:30.615360 15028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:30.616598 15028 hybrid_clock.cc:648] HybridClock initialized: now 1790219550616553 us; error 38 us; skew 500 ppm
I20260924 03:12:30.616942 15028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:30 dist-test-slave-5l98 krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:30.620018 15028 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:12:30.621757 15028 webserver.cc:460] Webserver started at http://127.0.97.2:36163/ using document root <none> and password file <none>
I20260924 03:12:30.622470 15028 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:30.622627 15028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:30.628032 15028 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 03:12:30.631247 15043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:30.632830 15028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:12:30.633033 15028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "42041e2066ab4a6d878e1e2a374d5301"
format_stamp: "Formatted at 2026-09-24 03:12:27 on dist-test-slave-5l98"
server_key: "cb8467d9ff0d36dc898c645a0bd6c88c"
server_key_iv: "5669df9398b1f71059ac9823ad3e7f03"
server_key_version: "encryptionkey@0"
I20260924 03:12:30.634065 15028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:30.664465 15028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:30.669591 15028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:30.669916 15028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:30.671049 15028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:30.672596 15028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:30.672742 15028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:30.672895 15028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:30.672981 15028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:30.710717 15028 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33695
I20260924 03:12:30.710762 15155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33695 every 8 connection(s)
I20260924 03:12:30.712615 15028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:12:30.720434 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 15028
I20260924 03:12:30.721199 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:38731
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41411
--webserver_port=35959
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:12:30 dist-test-slave-5l98 krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:12:30.737362 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:30.715537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59391 (local address 127.0.97.62:41411)
0924 03:12:30.715737 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:30.715742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:30.717010 (+ 1268us) server_negotiation.cc:407] Connection header received
0924 03:12:30.718845 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:30.718851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:30.718983 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:30.719096 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:30.720951 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:30.721866 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:30.724592 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:30.725021 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:30.725127 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:30.729622 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:30.729627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:30.729632 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:30.729672 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:30.733348 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:30.734132 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:30.734142 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:30.734149 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:30.734319 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:30.734880 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:30.734888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:30.734895 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:30.735351 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:30.735521 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:30.736527 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0924 03:12:30.736818 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 03:12:30.738173 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:30.715945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.2:59391)
0924 03:12:30.716822 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:30.716866 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:30.718567 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:30.719337 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:30.719346 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:30.719888 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:30.720747 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:30.720773 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:30.722055 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:30.722063 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:30.724312 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:30.724343 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:30.725272 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:30.725282 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:30.725626 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:30.726406 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:30.726441 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:30.729307 (+ 2866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:30.733521 (+ 4214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:30.733529 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:30.733543 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:30.733976 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:30.734474 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:30.734478 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:30.734481 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:30.734721 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:30.735678 (+ 957us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:30.735685 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:30.736054 (+ 369us) client_negotiation.cc:769] Sending connection context
0924 03:12:30.736314 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 03:12:30.736903 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":310,"threads_started":1}
I20260924 03:12:30.739786 15156 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41411
I20260924 03:12:30.740113 15156 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:30.740891 15156 heartbeater.cc:511] Master 127.0.97.62:41411 requested a full tablet report, sending...
I20260924 03:12:30.742333 14839 ts_manager.cc:194] Registered new tserver with Master: 42041e2066ab4a6d878e1e2a374d5301 (127.0.97.2:33695)
I20260924 03:12:30.743148 14839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59391
W20260924 03:12:31.060286 15159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:31.060652 15159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:31.060833 15159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:31.068244 15159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:31.068410 15159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:31.068624 15159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:12:31.078598 15159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:38731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35959
--tserver_master_addrs=127.0.97.62:41411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:31.079959 15159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:31.081360 15159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:31.094761 15164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:31.094761 15165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:31.095043 15167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:31.096333 15159 server_base.cc:1034] running on GCE node
I20260924 03:12:31.097040 15159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:31.097970 15159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:31.099188 15159 hybrid_clock.cc:648] HybridClock initialized: now 1790219551099151 us; error 37 us; skew 500 ppm
I20260924 03:12:31.099469 15159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:31 dist-test-slave-5l98 krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:31.102864 15159 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:12:31.104714 15159 webserver.cc:460] Webserver started at http://127.0.97.3:35959/ using document root <none> and password file <none>
I20260924 03:12:31.105460 15159 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:31.105631 15159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:31.111681 15159 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 03:12:31.115057 15174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:31.116772 15159 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:12:31.116978 15159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "78f63d155fdc4959869cf5134eb15498"
format_stamp: "Formatted at 2026-09-24 03:12:28 on dist-test-slave-5l98"
server_key: "32c4add8bf937c5ba7b2250f553dcd53"
server_key_iv: "ab99e99ab4673ec8f5b2a1c9b42bb492"
server_key_version: "encryptionkey@0"
I20260924 03:12:31.117972 15159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:31.146967 15159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:31.151728 15159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:31.151963 15159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:31.153047 15159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:31.154471 15159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:31.154587 15159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:31.154693 15159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:31.154753 15159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:31.190297 15159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38731
I20260924 03:12:31.190420 15286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38731 every 8 connection(s)
I20260924 03:12:31.191829 15159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:12:31.201105 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 15159
Sep 24 03:12:31 dist-test-slave-5l98 krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:12:31.217188 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:31.194748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38423 (local address 127.0.97.62:41411)
0924 03:12:31.194971 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:31.194978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:31.196245 (+ 1267us) server_negotiation.cc:407] Connection header received
0924 03:12:31.198190 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:31.198195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:31.198270 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:31.198369 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:31.200016 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.201224 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.203319 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.204152 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.204309 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:31.208760 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:31.208765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:31.208770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:31.208804 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:31.213051 (+ 4247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:31.214255 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:31.214262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:31.214268 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:31.214434 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:31.215114 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:31.215118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:31.215120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:31.215671 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:31.215835 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:31.216745 (+ 910us) server_negotiation.cc:299] Negotiation successful
0924 03:12:31.216944 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 03:12:31.218869 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:31.195265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.3:38423)
0924 03:12:31.196076 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:31.196113 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:31.197899 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:31.198697 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:31.198706 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:31.199251 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:31.199811 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:31.199828 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.201433 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.201438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:31.203119 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:31.203137 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.204397 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.204402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:31.204727 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:31.205498 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:31.205613 (+ 115us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:31.208484 (+ 2871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:31.213238 (+ 4754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:31.213250 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:31.213272 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:31.214065 (+ 793us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:31.214640 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:31.214647 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:31.214650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:31.214910 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:31.215831 (+ 921us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:31.215838 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:31.216231 (+ 393us) client_negotiation.cc:769] Sending connection context
0924 03:12:31.216663 (+ 432us) client_negotiation.cc:240] Negotiation successful
0924 03:12:31.217326 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":345,"threads_started":1}
I20260924 03:12:31.219833 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:31.202165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36874 (local address 127.0.97.62:41411)
0924 03:12:31.202676 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:31.202682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:31.203128 (+ 446us) server_negotiation.cc:407] Connection header received
0924 03:12:31.203273 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:31.203279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:31.203363 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:31.203452 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:31.205507 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.206656 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.208545 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.209104 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.209206 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:31.213280 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:31.213287 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:31.213292 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:31.213332 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:31.216655 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:31.217550 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:31.217554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:31.217557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:31.217674 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:31.218115 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:31.218117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:31.218119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:31.218377 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:31.218634 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:31.219206 (+ 572us) server_negotiation.cc:299] Negotiation successful
0924 03:12:31.219413 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":267,"threads_started":1}
I20260924 03:12:31.220651 15287 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41411
I20260924 03:12:31.221113 15287 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:31.222193 15287 heartbeater.cc:511] Master 127.0.97.62:41411 requested a full tablet report, sending...
I20260924 03:12:31.223917 14839 ts_manager.cc:194] Registered new tserver with Master: 78f63d155fdc4959869cf5134eb15498 (127.0.97.3:38731)
I20260924 03:12:31.225211 14839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38423
I20260924 03:12:31.236106 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:12:31.244113 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:31.237534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36890 (local address 127.0.97.62:41411)
0924 03:12:31.237768 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:31.237774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:31.238009 (+ 235us) server_negotiation.cc:407] Connection header received
0924 03:12:31.238214 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:31.238218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:31.238363 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:31.238537 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:12:31.239227 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.240030 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.241641 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.242103 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.242199 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:31.242694 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:12:31.243256 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:12:31.243373 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:31.243630 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 03:12:31.243830 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 03:12:31.245285 14839 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:36890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:12:31.253269 14839 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:12:31.264195 15024 heartbeater.cc:503] Master 127.0.97.62:41411 was elected leader, sending a full tablet report...
I20260924 03:12:31.277519 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:31.266496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.0.97.2:33695)
0924 03:12:31.268602 (+ 2106us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:31.268613 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:31.268650 (+ 37us) server_negotiation.cc:407] Connection header received
0924 03:12:31.268758 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:31.268767 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:31.269041 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:31.269256 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:31.270028 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.271593 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.274813 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.276081 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.276367 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:31.276635 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:31.277008 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 03:12:31.277183 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1926,"thread_start_us":1823,"threads_started":1}
I20260924 03:12:31.277711 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:31.265830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33695 (local address 127.0.0.1:33712)
0924 03:12:31.266979 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:31.267016 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:31.267176 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:31.269313 (+ 2137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:31.269321 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:31.269373 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:31.269819 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:31.269834 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.271803 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.271812 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:31.274587 (+ 2775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:31.274602 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.276440 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.276450 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:31.276702 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:31.276758 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 03:12:31.276916 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 03:12:31.277130 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"mutex_wait_us":2,"thread_start_us":875,"threads_started":1}
I20260924 03:12:31.279826 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:31.266631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50194 (local address 127.0.97.3:38731)
0924 03:12:31.269012 (+ 2381us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:31.269020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:31.269647 (+ 627us) server_negotiation.cc:407] Connection header received
0924 03:12:31.269957 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:31.269967 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:31.270734 (+ 767us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:31.270917 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:31.272239 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.273763 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.276593 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.278391 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.278482 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:31.278698 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:31.279296 (+ 598us) server_negotiation.cc:299] Negotiation successful
0924 03:12:31.279491 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2223,"thread_start_us":2051,"threads_started":1}
I20260924 03:12:31.279876 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:31.266216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38731 (local address 127.0.0.1:50194)
0924 03:12:31.269515 (+ 3299us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:31.269533 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:31.269758 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:31.271435 (+ 1677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:31.271440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:31.271465 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:31.271990 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:31.272017 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.274238 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.274243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:31.276354 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:31.276370 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.278907 (+ 2537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.278913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:31.279090 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:31.279126 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 03:12:31.279223 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 03:12:31.279401 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3154,"mutex_wait_us":4,"thread_start_us":2989,"threads_started":1}
I20260924 03:12:31.280709 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:31.269884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42722 (local address 127.0.97.1:45013)
0924 03:12:31.270533 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:31.270544 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:31.272456 (+ 1912us) server_negotiation.cc:407] Connection header received
0924 03:12:31.272679 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:31.272687 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:31.272912 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:31.273120 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:31.273792 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.275319 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.277826 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.279292 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.279415 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:31.279648 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:31.280196 (+ 548us) server_negotiation.cc:299] Negotiation successful
0924 03:12:31.280376 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":291,"threads_started":1}
I20260924 03:12:31.280714 15302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:31.269644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45013 (local address 127.0.0.1:42722)
0924 03:12:31.272324 (+ 2680us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:31.272365 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:31.272526 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:31.273104 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:31.273109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:31.273134 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:31.273602 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:31.273613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.275462 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.275467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:31.277632 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:31.277654 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.279805 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.279813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:31.280017 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:31.280047 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 03:12:31.280137 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 03:12:31.280280 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2537,"mutex_wait_us":30,"thread_start_us":2327,"threads_started":1}
I20260924 03:12:31.282217 15091 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf2d96397dd4e9281ae54958ded53fd (DEFAULT_TABLE table=test-table [id=0c75048017da4df9addebdedee96c248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:31.283694 15091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf2d96397dd4e9281ae54958ded53fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:31.283784 15222 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf2d96397dd4e9281ae54958ded53fd (DEFAULT_TABLE table=test-table [id=0c75048017da4df9addebdedee96c248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:31.283928 14959 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf2d96397dd4e9281ae54958ded53fd (DEFAULT_TABLE table=test-table [id=0c75048017da4df9addebdedee96c248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:31.285127 14959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf2d96397dd4e9281ae54958ded53fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:31.285123 15222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf2d96397dd4e9281ae54958ded53fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:31.297271 15303 tablet_bootstrap.cc:492] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2: Bootstrap starting.
I20260924 03:12:31.299712 15304 tablet_bootstrap.cc:492] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301: Bootstrap starting.
I20260924 03:12:31.301034 15303 tablet_bootstrap.cc:654] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:31.301301 15305 tablet_bootstrap.cc:492] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498: Bootstrap starting.
I20260924 03:12:31.303025 15303 log.cc:824] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:31.303297 15304 tablet_bootstrap.cc:654] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:31.304405 15304 log.cc:824] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:31.305186 15305 tablet_bootstrap.cc:654] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:31.306221 15305 log.cc:824] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:31.306437 15303 tablet_bootstrap.cc:492] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2: No bootstrap required, opened a new log
I20260924 03:12:31.306728 15303 ts_tablet_manager.cc:1397] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 03:12:31.307359 15304 tablet_bootstrap.cc:492] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301: No bootstrap required, opened a new log
I20260924 03:12:31.307614 15304 ts_tablet_manager.cc:1397] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260924 03:12:31.308930 15305 tablet_bootstrap.cc:492] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498: No bootstrap required, opened a new log
I20260924 03:12:31.309182 15305 ts_tablet_manager.cc:1397] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 03:12:31.317184 15305 raft_consensus.cc:348] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } } peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } }
I20260924 03:12:31.317638 15305 raft_consensus.cc:374] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:31.317734 15305 raft_consensus.cc:729] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f63d155fdc4959869cf5134eb15498, State: Initialized, Role: FOLLOWER
I20260924 03:12:31.317675 15303 raft_consensus.cc:348] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } } peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } }
I20260924 03:12:31.318128 15303 raft_consensus.cc:374] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:31.318225 15303 raft_consensus.cc:729] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a953cfe71cb49dda86b666b75961cc2, State: Initialized, Role: FOLLOWER
I20260924 03:12:31.318230 15304 raft_consensus.cc:348] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } } peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } }
I20260924 03:12:31.318655 15304 raft_consensus.cc:374] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:31.318454 15305 consensus_queue.cc:260] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } } peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } }
I20260924 03:12:31.318754 15304 raft_consensus.cc:729] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42041e2066ab4a6d878e1e2a374d5301, State: Initialized, Role: FOLLOWER
I20260924 03:12:31.318812 15303 consensus_queue.cc:260] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } } peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } }
I20260924 03:12:31.319427 15304 consensus_queue.cc:260] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } } peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } }
I20260924 03:12:31.319780 15287 heartbeater.cc:503] Master 127.0.97.62:41411 was elected leader, sending a full tablet report...
I20260924 03:12:31.320358 15305 ts_tablet_manager.cc:1428] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260924 03:12:31.321815 15303 ts_tablet_manager.cc:1428] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260924 03:12:31.322072 15304 ts_tablet_manager.cc:1428] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
I20260924 03:12:31.322613 15156 heartbeater.cc:503] Master 127.0.97.62:41411 was elected leader, sending a full tablet report...
W20260924 03:12:31.443809 15288 tablet.cc:2369] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:12:31.465251 15157 tablet.cc:2369] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:12:31.479454 15025 tablet.cc:2369] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:12:31.557440 15309 raft_consensus.cc:482] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:12:31.557788 15309 raft_consensus.cc:504] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } } peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } }
I20260924 03:12:31.559598 15309 leader_election.cc:288] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42041e2066ab4a6d878e1e2a374d5301 (127.0.97.2:33695), 2a953cfe71cb49dda86b666b75961cc2 (127.0.97.1:45013)
I20260924 03:12:31.566835 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:31.560215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50003 (local address 127.0.97.1:45013)
0924 03:12:31.560596 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:31.560603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:31.560625 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:12:31.560796 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:31.560805 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:31.560865 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:31.560964 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:31.561402 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.562538 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.564568 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.565881 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.566000 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:31.566080 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:31.566448 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 03:12:31.566641 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260924 03:12:31.566845 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:31.560141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45013 (local address 127.0.97.3:50003)
0924 03:12:31.560414 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:31.560435 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:31.560616 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:31.560991 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:31.560996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:31.561022 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:31.561213 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:31.561221 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.562676 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.562682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:31.564357 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:31.564370 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.566050 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.566057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:31.566266 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:31.566305 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 03:12:31.566416 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 03:12:31.566558 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260924 03:12:31.567922 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:31.560176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44795 (local address 127.0.97.2:33695)
0924 03:12:31.560470 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:31.560479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:31.561291 (+ 812us) server_negotiation.cc:407] Connection header received
0924 03:12:31.561473 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:31.561482 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:31.561540 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:31.561608 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:31.562235 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.563200 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.565688 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.566985 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.567118 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:31.567230 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:31.567639 (+ 409us) server_negotiation.cc:299] Negotiation successful
0924 03:12:31.567740 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260924 03:12:31.568202 15312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:31.560212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33695 (local address 127.0.97.3:44795)
0924 03:12:31.561174 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:31.561190 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:31.561329 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:31.561660 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:31.561665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:31.561693 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:31.562063 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:31.562074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.563423 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.563427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:31.565494 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:31.565512 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.567256 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.567262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:31.567439 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:31.567471 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 03:12:31.567611 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 03:12:31.567779 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":4,"thread_start_us":574,"threads_started":1}
I20260924 03:12:31.568136 14979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf2d96397dd4e9281ae54958ded53fd" candidate_uuid: "78f63d155fdc4959869cf5134eb15498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a953cfe71cb49dda86b666b75961cc2" is_pre_election: true
I20260924 03:12:31.568863 14979 raft_consensus.cc:2454] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78f63d155fdc4959869cf5134eb15498 in term 0.
I20260924 03:12:31.569419 15111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf2d96397dd4e9281ae54958ded53fd" candidate_uuid: "78f63d155fdc4959869cf5134eb15498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42041e2066ab4a6d878e1e2a374d5301" is_pre_election: true
I20260924 03:12:31.569768 15176 leader_election.cc:302] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a953cfe71cb49dda86b666b75961cc2, 78f63d155fdc4959869cf5134eb15498; no voters:
I20260924 03:12:31.569976 15111 raft_consensus.cc:2454] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78f63d155fdc4959869cf5134eb15498 in term 0.
I20260924 03:12:31.570283 15309 raft_consensus.cc:2781] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:12:31.570472 15309 raft_consensus.cc:482] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:12:31.570616 15309 raft_consensus.cc:3037] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:31.573446 15309 raft_consensus.cc:504] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } } peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } }
I20260924 03:12:31.574132 15309 leader_election.cc:288] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 [CANDIDATE]: Term 1 election: Requested vote from peers 42041e2066ab4a6d878e1e2a374d5301 (127.0.97.2:33695), 2a953cfe71cb49dda86b666b75961cc2 (127.0.97.1:45013)
I20260924 03:12:31.574613 15111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf2d96397dd4e9281ae54958ded53fd" candidate_uuid: "78f63d155fdc4959869cf5134eb15498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42041e2066ab4a6d878e1e2a374d5301"
I20260924 03:12:31.574788 15111 raft_consensus.cc:3037] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:31.574756 14979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf2d96397dd4e9281ae54958ded53fd" candidate_uuid: "78f63d155fdc4959869cf5134eb15498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a953cfe71cb49dda86b666b75961cc2"
I20260924 03:12:31.574995 14979 raft_consensus.cc:3037] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:31.577067 15111 raft_consensus.cc:2454] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78f63d155fdc4959869cf5134eb15498 in term 1.
I20260924 03:12:31.577515 15178 leader_election.cc:302] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42041e2066ab4a6d878e1e2a374d5301, 78f63d155fdc4959869cf5134eb15498; no voters:
I20260924 03:12:31.577711 14979 raft_consensus.cc:2454] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78f63d155fdc4959869cf5134eb15498 in term 1.
I20260924 03:12:31.577937 15309 raft_consensus.cc:2781] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:31.578591 15309 raft_consensus.cc:686] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 [term 1 LEADER]: Becoming Leader. State: Replica: 78f63d155fdc4959869cf5134eb15498, State: Running, Role: LEADER
I20260924 03:12:31.579071 15309 consensus_queue.cc:237] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } } peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } }
I20260924 03:12:31.584041 14836 catalog_manager.cc:5495] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78f63d155fdc4959869cf5134eb15498 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "78f63d155fdc4959869cf5134eb15498" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } health_report { overall_health: UNKNOWN } } }
I20260924 03:12:31.653944 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:31.648438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50196 (local address 127.0.97.3:38731)
0924 03:12:31.648698 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:31.648702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:31.648721 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:12:31.648911 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:31.648915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:31.649008 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:31.649190 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:12:31.649532 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.650294 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.651769 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.652154 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.652245 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:31.652796 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:12:31.653332 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:12:31.653431 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:31.653678 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 03:12:31.653745 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260924 03:12:31.662410 14979 raft_consensus.cc:1261] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Refusing update from remote peer 78f63d155fdc4959869cf5134eb15498: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:31.662407 15111 raft_consensus.cc:1261] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301 [term 1 FOLLOWER]: Refusing update from remote peer 78f63d155fdc4959869cf5134eb15498: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:31.663601 15309 consensus_queue.cc:1035] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:31.663976 15313 consensus_queue.cc:1035] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:31.679956 15318 mvcc.cc:205] Tried to move back new op lower bound from 7332739283597729792 to 7332739283271385088. Current Snapshot: MvccSnapshot[applied={T|T < 7332739283597729792}]
I20260924 03:12:31.692893 14837 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:36890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:12:31.693202 14837 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:36890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:12:31.697224 14837 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c: Sending DeleteTablet for 3 replicas of tablet 2bf2d96397dd4e9281ae54958ded53fd
I20260924 03:12:31.698477 15222 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bf2d96397dd4e9281ae54958ded53fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50194
I20260924 03:12:31.698570 15091 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bf2d96397dd4e9281ae54958ded53fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33712
I20260924 03:12:31.700394 14959 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bf2d96397dd4e9281ae54958ded53fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42722
I20260924 03:12:31.708652 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 14876
I20260924 03:12:31.730720 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:31.709969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.0.97.1:45013)
0924 03:12:31.710585 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:31.710591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:31.710609 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:12:31.710670 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:31.710675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:31.710780 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:31.710888 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:31.712928 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.714072 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.716223 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.717049 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.717189 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:31.721424 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:31.721451 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:31.721474 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:31.721529 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:31.726324 (+ 4795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:31.727453 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:31.727464 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:31.727468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:31.727654 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:31.728007 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:31.728013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:31.728017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:31.728568 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:31.728878 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:31.730039 (+ 1161us) server_negotiation.cc:299] Negotiation successful
0924 03:12:31.730283 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494}
I20260924 03:12:31.733428 15326 tablet_replica.cc:330] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301: stopping tablet replica
I20260924 03:12:31.733979 15326 raft_consensus.cc:2229] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:12:31.734501 15326 raft_consensus.cc:2258] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:31.737313 15019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:31.736567 15326 ts_tablet_manager.cc:1905] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:31.749078 15327 tablet_replica.cc:330] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498: stopping tablet replica
I20260924 03:12:31.749651 15327 raft_consensus.cc:2229] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:12:31.750368 15327 raft_consensus.cc:2258] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:31.752460 15327 ts_tablet_manager.cc:1905] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:31.753480 15326 ts_tablet_manager.cc:1918] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:31.753702 15326 log.cc:1196] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/2bf2d96397dd4e9281ae54958ded53fd
I20260924 03:12:31.754328 15326 ts_tablet_manager.cc:1939] T 2bf2d96397dd4e9281ae54958ded53fd P 42041e2066ab4a6d878e1e2a374d5301: Deleting consensus metadata
I20260924 03:12:31.756433 14826 catalog_manager.cc:4841] TS 42041e2066ab4a6d878e1e2a374d5301 (127.0.97.2:33695): tablet 2bf2d96397dd4e9281ae54958ded53fd (table test-table [id=0c75048017da4df9addebdedee96c248]) successfully deleted
I20260924 03:12:31.759112 15327 ts_tablet_manager.cc:1918] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:31.759325 15327 log.cc:1196] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/2bf2d96397dd4e9281ae54958ded53fd
I20260924 03:12:31.759833 15327 ts_tablet_manager.cc:1939] T 2bf2d96397dd4e9281ae54958ded53fd P 78f63d155fdc4959869cf5134eb15498: Deleting consensus metadata
I20260924 03:12:31.761402 14824 catalog_manager.cc:4841] TS 78f63d155fdc4959869cf5134eb15498 (127.0.97.3:38731): tablet 2bf2d96397dd4e9281ae54958ded53fd (table test-table [id=0c75048017da4df9addebdedee96c248]) successfully deleted
I20260924 03:12:31.875262 15328 tablet_replica.cc:330] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2: stopping tablet replica
I20260924 03:12:31.875784 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 14876
I20260924 03:12:31.875800 15328 raft_consensus.cc:2229] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 03:12:31.899983 14824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45013 (error 108)
I20260924 03:12:31.900270 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 15028
I20260924 03:12:31.902822 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:31.901949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45013 (local address 127.0.0.1:42748)
0924 03:12:31.902436 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:31.902571 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":229,"threads_started":1}
W20260924 03:12:31.903133 14824 catalog_manager.cc:4584] TS 2a953cfe71cb49dda86b666b75961cc2 (127.0.97.1:45013): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2bf2d96397dd4e9281ae54958ded53fd: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45013: connect: Connection refused (error 111)
I20260924 03:12:31.914973 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:31.901293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33722 (local address 127.0.97.2:33695)
0924 03:12:31.901479 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:31.901485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:31.901507 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:12:31.901701 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:31.901709 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:31.901819 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:31.901943 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:31.903733 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.904460 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.905946 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:31.906370 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:31.906508 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:31.909191 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:31.909204 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:31.909219 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:31.909252 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:31.912025 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:31.912839 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:31.912854 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:31.912856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:31.912985 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:31.913389 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:31.913399 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:31.913401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:31.913773 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:31.914083 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:31.914403 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 03:12:31.914625 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:12:31.915743 15151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:31.956791 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:31.956240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45013 (local address 127.0.0.1:42750)
0924 03:12:31.956465 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:31.956612 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 03:12:31.957969 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:31.957553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45013 (local address 127.0.0.1:42758)
0924 03:12:31.957733 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:31.957817 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 03:12:32.008720 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 15028
I20260924 03:12:32.018690 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:32.018090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45013 (local address 127.0.0.1:42766)
0924 03:12:32.018362 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:32.018482 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 03:12:32.019966 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:32.019518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45013 (local address 127.0.0.1:42776)
0924 03:12:32.019699 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:32.019795 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 03:12:32.027509 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 15159
I20260924 03:12:32.042703 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:32.028279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50208 (local address 127.0.97.3:38731)
0924 03:12:32.028444 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:32.028450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:32.028577 (+ 127us) server_negotiation.cc:407] Connection header received
0924 03:12:32.028809 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:32.028815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:32.028895 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:32.029016 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:32.030402 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:32.031213 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:32.032804 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:32.033273 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:32.033433 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:32.036074 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:32.036084 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:32.036099 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:32.036137 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:32.039544 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:32.040387 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:32.040395 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:32.040401 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:32.040640 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:32.041056 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:32.041061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:32.041065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:32.041442 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:32.041797 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:32.042086 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 03:12:32.042312 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:12:32.043450 15282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:32.101006 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:32.100414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45013 (local address 127.0.0.1:42782)
0924 03:12:32.100717 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:32.100825 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260924 03:12:32.102222 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:32.101755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45013 (local address 127.0.0.1:42792)
0924 03:12:32.101988 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:32.102072 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260924 03:12:32.172387 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 15159
I20260924 03:12:32.193449 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 14806
I20260924 03:12:32.195417 14869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:32.320475 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:32.319576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45013 (local address 127.0.0.1:42796)
0924 03:12:32.319879 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:32.320022 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 03:12:32.320544 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 14806
I20260924 03:12:32.342669 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41411
--webserver_interface=127.0.97.62
--webserver_port=42705
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:32.684949 15335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:32.685266 15335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:32.685423 15335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:32.692965 15335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:32.693092 15335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:32.693212 15335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:32.693318 15335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:32.693405 15335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:32.704012 15335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:32.705524 15335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:32.706836 15335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:32.720917 15340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:32.720947 15341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:32.720897 15343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:32.721870 15335 server_base.cc:1034] running on GCE node
I20260924 03:12:32.722692 15335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:32.724144 15335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:32.725379 15335 hybrid_clock.cc:648] HybridClock initialized: now 1790219552725333 us; error 39 us; skew 500 ppm
I20260924 03:12:32.725745 15335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:32 dist-test-slave-5l98 krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:32.728907 15335 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:12:32.730662 15335 webserver.cc:460] Webserver started at http://127.0.97.62:42705/ using document root <none> and password file <none>
I20260924 03:12:32.731448 15335 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:32.731604 15335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:32.737357 15335 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:12:32.740983 15350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:32.742408 15335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:12:32.742620 15335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "fb2f76d2322f495bb8b176130b72923c"
format_stamp: "Formatted at 2026-09-24 03:12:26 on dist-test-slave-5l98"
server_key: "c1e7ca83d70edb168a8f8f1618288bb9"
server_key_iv: "fe0301ee266894cfa84c792cb107235c"
server_key_version: "encryptionkey@0"
I20260924 03:12:32.743625 15335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:32.762773 15335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:32.767531 15335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:32.767853 15335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:32.787894 15335 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41411
I20260924 03:12:32.787928 15401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41411 every 8 connection(s)
I20260924 03:12:32.789538 15335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:32.796358 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 15335
I20260924 03:12:32.796938 15402 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:12:32.797423 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45013
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41411
--webserver_port=43533
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:12:32.802414 15402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c: Bootstrap starting.
I20260924 03:12:32.826591 15402 log.cc:824] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:32.844233 15402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:12:32.844843 15402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c: Bootstrap complete.
I20260924 03:12:32.855216 15402 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2f76d2322f495bb8b176130b72923c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } }
I20260924 03:12:32.856272 15402 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb2f76d2322f495bb8b176130b72923c, State: Initialized, Role: FOLLOWER
I20260924 03:12:32.856930 15402 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2f76d2322f495bb8b176130b72923c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } }
I20260924 03:12:32.857172 15402 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:32.857312 15402 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:32.857621 15402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 2 FOLLOWER]: Advancing to term 3
I20260924 03:12:32.861593 15402 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2f76d2322f495bb8b176130b72923c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } }
I20260924 03:12:32.862133 15402 leader_election.cc:302] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb2f76d2322f495bb8b176130b72923c; no voters:
I20260924 03:12:32.862825 15402 leader_election.cc:288] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 03:12:32.862975 15406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 3 FOLLOWER]: Leader election won for term 3
I20260924 03:12:32.864053 15406 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [term 3 LEADER]: Becoming Leader. State: Replica: fb2f76d2322f495bb8b176130b72923c, State: Running, Role: LEADER
I20260924 03:12:32.864691 15406 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2f76d2322f495bb8b176130b72923c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } }
I20260924 03:12:32.865156 15402 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:32.867759 15408 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb2f76d2322f495bb8b176130b72923c. Latest consensus state: current_term: 3 leader_uuid: "fb2f76d2322f495bb8b176130b72923c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2f76d2322f495bb8b176130b72923c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } } }
I20260924 03:12:32.868134 15408 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:32.867877 15407 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fb2f76d2322f495bb8b176130b72923c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2f76d2322f495bb8b176130b72923c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } } }
I20260924 03:12:32.868252 15407 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2f76d2322f495bb8b176130b72923c [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:32.878077 15411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:32.889479 15411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:32.891782 15411 catalog_manager.cc:1259] Loaded cluster ID: 3d744971c4ca4c908e14f06ecdb9513f
I20260924 03:12:32.891928 15411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 03:12:32.898552 15411 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f04dafaf9ad google::LogMessage::Fail() at ??:0
@ 0x7f04dafb288a google::LogMessage::SendToLog() at ??:0
@ 0x7f04dafaf4de google::LogMessage::Flush() at ??:0
@ 0x7f04dafb0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f04f09e2512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f04f09df976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f04f0a67610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f04f0a67439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f04ddbfb345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f04ddbfe91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f04ddbfe799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f04ddbd9eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f04e0eb36db start_thread at ??:0
@ 0x7f04d9e4961f clone at ??:0
W20260924 03:12:33.182729 15404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:33.183069 15404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:33.183241 15404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:33.191030 15404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:33.191212 15404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:33.191397 15404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:12:33.201853 15404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43533
--tserver_master_addrs=127.0.97.62:41411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:33.203264 15404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:33.204646 15404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:33.218359 15444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:33.218456 15443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:33.219089 15404 server_base.cc:1034] running on GCE node
W20260924 03:12:33.219179 15446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:33.219834 15404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:33.220867 15404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:33.222100 15404 hybrid_clock.cc:648] HybridClock initialized: now 1790219553222045 us; error 43 us; skew 500 ppm
I20260924 03:12:33.222412 15404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:33 dist-test-slave-5l98 krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:33.225747 15404 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:12:33.227316 15404 webserver.cc:460] Webserver started at http://127.0.97.1:43533/ using document root <none> and password file <none>
I20260924 03:12:33.228051 15404 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:33.228206 15404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:33.234184 15404 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 03:12:33.237162 15453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:33.238565 15404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:12:33.238765 15404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "2a953cfe71cb49dda86b666b75961cc2"
format_stamp: "Formatted at 2026-09-24 03:12:27 on dist-test-slave-5l98"
server_key: "03a7a4305fa28a3175d0b82caf568269"
server_key_iv: "241282bf6e12cfc8fac99b9631cd918f"
server_key_version: "encryptionkey@0"
I20260924 03:12:33.239778 15404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:33.255741 15404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:33.259985 15404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:33.260288 15404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:33.261364 15404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:33.263314 15460 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 03:12:33.269074 15404 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 03:12:33.269227 15404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260924 03:12:33.269387 15404 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 03:12:33.272593 15404 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 03:12:33.272702 15404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.004s
I20260924 03:12:33.273237 15460 tablet_bootstrap.cc:492] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2: Bootstrap starting.
I20260924 03:12:33.301147 15460 log.cc:824] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:33.314807 15460 tablet_bootstrap.cc:492] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:12:33.315171 15404 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45013
I20260924 03:12:33.315213 15567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45013 every 8 connection(s)
I20260924 03:12:33.315513 15460 tablet_bootstrap.cc:492] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2: Bootstrap complete.
I20260924 03:12:33.316421 15404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:12:33.316883 15460 ts_tablet_manager.cc:1397] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2: Time spent bootstrapping tablet: real 0.044s user 0.016s sys 0.026s
I20260924 03:12:33.321900 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:33.319790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:49861)
0924 03:12:33.320737 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:33.321001 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":424,"threads_started":1}
W20260924 03:12:33.322638 15568 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41411 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41411: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
I20260924 03:12:33.323547 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:33.323177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:34289)
0924 03:12:33.323356 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:33.323428 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 03:12:33.324548 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:33.324120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:35743)
0924 03:12:33.324262 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:33.324348 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 03:12:33.325106 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 15404
I20260924 03:12:33.325893 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33695
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41411
--webserver_port=36163
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:12:33.326189 15460 raft_consensus.cc:348] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } } peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } }
I20260924 03:12:33.327623 15460 raft_consensus.cc:729] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a953cfe71cb49dda86b666b75961cc2, State: Initialized, Role: FOLLOWER
W20260924 03:12:33.327811 15568 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:12:33.328275 15460 consensus_queue.cc:260] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } } peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } }
I20260924 03:12:33.330026 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:33.329654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:45467)
0924 03:12:33.329812 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:33.329913 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 03:12:33.330053 15460 ts_tablet_manager.cc:1428] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260924 03:12:33.331029 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:33.330587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:42089)
0924 03:12:33.330732 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:33.330861 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 03:12:33.659431 15572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:33.659775 15572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:33.659955 15572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:33.667346 15572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:33.667518 15572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:33.667735 15572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:12:33.677874 15572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36163
--tserver_master_addrs=127.0.97.62:41411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:33.679251 15572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:33.680634 15572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:33.694536 15577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:33.694483 15580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:33.694500 15578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:33.695791 15572 server_base.cc:1034] running on GCE node
I20260924 03:12:33.696601 15572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:33.697515 15572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:33.698721 15572 hybrid_clock.cc:648] HybridClock initialized: now 1790219553698670 us; error 50 us; skew 500 ppm
I20260924 03:12:33.698998 15572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:33 dist-test-slave-5l98 krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:33.702190 15572 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:12:33.703821 15572 webserver.cc:460] Webserver started at http://127.0.97.2:36163/ using document root <none> and password file <none>
I20260924 03:12:33.704589 15572 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:33.704722 15572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:33.710157 15572 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:12:33.712898 15587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:33.714568 15572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:12:33.714757 15572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "42041e2066ab4a6d878e1e2a374d5301"
format_stamp: "Formatted at 2026-09-24 03:12:27 on dist-test-slave-5l98"
server_key: "cb8467d9ff0d36dc898c645a0bd6c88c"
server_key_iv: "5669df9398b1f71059ac9823ad3e7f03"
server_key_version: "encryptionkey@0"
I20260924 03:12:33.715848 15572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:33.731019 15572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:33.735313 15572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:33.735630 15572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:33.736763 15572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:33.738135 15572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:33.738260 15572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:33.738400 15572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:33.738492 15572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:33.772387 15572 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33695
I20260924 03:12:33.772434 15699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33695 every 8 connection(s)
I20260924 03:12:33.773880 15572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:12:33.778896 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:33.777110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.2:60021)
0924 03:12:33.777826 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:33.778017 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":271,"threads_started":1}
W20260924 03:12:33.779569 15700 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41411 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41411: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
I20260924 03:12:33.780418 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:33.780109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.2:37251)
0924 03:12:33.780251 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:33.780317 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 03:12:33.781306 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:33.781010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.2:39799)
0924 03:12:33.781136 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:33.781206 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 03:12:33.781548 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 15572
W20260924 03:12:33.781553 15700 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:12:33.782150 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:38731
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41411
--webserver_port=35959
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:34.105530 15703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:34.105799 15703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:34.105897 15703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:34.113055 15703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:34.113173 15703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:34.113282 15703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:12:34.123221 15703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:38731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35959
--tserver_master_addrs=127.0.97.62:41411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:34.124445 15703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:34.125689 15703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:34.138945 15711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:34.138947 15709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:34.138945 15708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:34.139726 15703 server_base.cc:1034] running on GCE node
I20260924 03:12:34.140393 15703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:34.141422 15703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:34.142674 15703 hybrid_clock.cc:648] HybridClock initialized: now 1790219554142611 us; error 53 us; skew 500 ppm
I20260924 03:12:34.143028 15703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:34 dist-test-slave-5l98 krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:34.146406 15703 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:12:34.147804 15703 webserver.cc:460] Webserver started at http://127.0.97.3:35959/ using document root <none> and password file <none>
I20260924 03:12:34.148458 15703 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:34.148638 15703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:34.154376 15703 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 03:12:34.157312 15718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:34.158684 15703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:12:34.158866 15703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "78f63d155fdc4959869cf5134eb15498"
format_stamp: "Formatted at 2026-09-24 03:12:28 on dist-test-slave-5l98"
server_key: "32c4add8bf937c5ba7b2250f553dcd53"
server_key_iv: "ab99e99ab4673ec8f5b2a1c9b42bb492"
server_key_version: "encryptionkey@0"
I20260924 03:12:34.159885 15703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:34.176999 15703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:34.181447 15703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:34.181775 15703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:34.183017 15703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:34.184618 15703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:34.184710 15703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:34.184880 15703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:34.184980 15703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:34.227068 15703 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38731
I20260924 03:12:34.227138 15830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38731 every 8 connection(s)
I20260924 03:12:34.228734 15703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:12:34.233755 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:34.231783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.3:45515)
0924 03:12:34.232682 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:34.232898 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":304,"threads_started":1}
W20260924 03:12:34.234593 15831 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41411 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41411: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
I20260924 03:12:34.235072 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 15703
I20260924 03:12:34.235517 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:34.235156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.3:45145)
0924 03:12:34.235333 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:34.235411 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 03:12:34.236527 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:34.236079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.3:50933)
0924 03:12:34.236267 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:34.236371 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260924 03:12:34.236788 15831 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:12:34.332756 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:34.331906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:39827)
0924 03:12:34.332385 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:34.332546 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":229,"threads_started":1}
I20260924 03:12:34.585641 15836 raft_consensus.cc:482] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:12:34.585956 15836 raft_consensus.cc:504] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } } peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } }
I20260924 03:12:34.587802 15836 leader_election.cc:288] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42041e2066ab4a6d878e1e2a374d5301 (127.0.97.2:33695), 78f63d155fdc4959869cf5134eb15498 (127.0.97.3:38731)
Sep 24 03:12:34 dist-test-slave-5l98 krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
Sep 24 03:12:34 dist-test-slave-5l98 krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 03:12:34.613161 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:34.588384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38731 (local address 127.0.97.1:41161)
0924 03:12:34.589944 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:34.589965 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:34.590087 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:34.592730 (+ 2643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:34.592739 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:34.593556 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:34.594239 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:34.594259 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:34.595840 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:34.595847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:34.598307 (+ 2460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:34.598323 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:34.599837 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:34.599844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:34.600243 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:34.601285 (+ 1042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:34.601332 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:34.604778 (+ 3446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:34.609680 (+ 4902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:34.609684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:34.609689 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:34.610078 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:34.610658 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:34.610661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:34.610663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:34.610800 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:34.611692 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:34.611696 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:34.611834 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 03:12:34.612003 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 03:12:34.612676 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1397,"mutex_wait_us":407,"spinlock_wait_cycles":29440,"thread_start_us":1059,"threads_started":1}
I20260924 03:12:34.613260 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:34.588578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41161 (local address 127.0.97.3:38731)
0924 03:12:34.590629 (+ 2051us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:34.590638 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:34.590668 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:12:34.592057 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:34.592066 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:34.592370 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:34.592734 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:34.594436 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:34.595645 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:34.598489 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:34.599548 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:34.599737 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:34.605067 (+ 5330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:34.605081 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:34.605100 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:34.605146 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:34.609482 (+ 4336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:34.610225 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:34.610233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:34.610241 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:34.610409 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:34.610919 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:34.610923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:34.610927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:34.611546 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:34.611675 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:34.612378 (+ 703us) server_negotiation.cc:299] Negotiation successful
0924 03:12:34.612822 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1910,"thread_start_us":1772,"threads_started":1}
I20260924 03:12:34.613279 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:34.588256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44279 (local address 127.0.97.2:33695)
0924 03:12:34.590349 (+ 2093us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:34.590368 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:34.590409 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:12:34.591866 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:34.591875 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:34.592236 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:34.592568 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:34.594482 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:34.595739 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:34.598595 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:34.599548 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:34.599737 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:34.604741 (+ 5004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:34.604749 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:34.604764 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:34.604820 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:34.608956 (+ 4136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:34.609892 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:34.609899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:34.609904 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:34.610107 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:34.610544 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:34.610552 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:34.610558 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:34.611237 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:34.611425 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:34.612379 (+ 954us) server_negotiation.cc:299] Negotiation successful
0924 03:12:34.612854 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1869,"thread_start_us":1760,"threads_started":1}
I20260924 03:12:34.613188 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:34.587984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33695 (local address 127.0.97.1:44279)
0924 03:12:34.588324 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:34.588370 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:34.589081 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:34.592728 (+ 3647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:34.592738 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:34.593576 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:34.594256 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:34.594267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:34.595905 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:34.595910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:34.598307 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:34.598323 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:34.599844 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:34.599850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:34.600239 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:34.601285 (+ 1046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:34.601331 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:34.604347 (+ 3016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:34.609134 (+ 4787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:34.609142 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:34.609153 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:34.609710 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:34.610251 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:34.610255 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:34.610257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:34.610413 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:34.611368 (+ 955us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:34.611373 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:34.611691 (+ 318us) client_negotiation.cc:769] Sending connection context
0924 03:12:34.612003 (+ 312us) client_negotiation.cc:240] Negotiation successful
0924 03:12:34.612784 (+ 781us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":640,"spinlock_wait_cycles":26496}
W20260924 03:12:34.616645 15457 leader_election.cc:341] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42041e2066ab4a6d878e1e2a374d5301 (127.0.97.2:33695): Not found: Tablet not found: 2bf2d96397dd4e9281ae54958ded53fd
W20260924 03:12:34.616931 15455 leader_election.cc:341] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 78f63d155fdc4959869cf5134eb15498 (127.0.97.3:38731): Not found: Tablet not found: 2bf2d96397dd4e9281ae54958ded53fd
I20260924 03:12:34.617053 15455 leader_election.cc:302] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a953cfe71cb49dda86b666b75961cc2; no voters: 42041e2066ab4a6d878e1e2a374d5301, 78f63d155fdc4959869cf5134eb15498
I20260924 03:12:34.617411 15836 raft_consensus.cc:2726] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:12:34.782895 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:34.782075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.2:42817)
0924 03:12:34.782529 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:34.782648 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":213,"threads_started":1}
I20260924 03:12:35.238160 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:35.237340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.3:38727)
0924 03:12:35.237790 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:35.237962 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":235,"threads_started":1}
I20260924 03:12:35.334427 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:35.333704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:33357)
0924 03:12:35.334171 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:35.334258 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":244,"threads_started":1}
I20260924 03:12:35.784847 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:35.784027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.2:37853)
0924 03:12:35.784588 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:35.784690 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":266,"threads_started":1}
I20260924 03:12:36.240020 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:36.239247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.3:36341)
0924 03:12:36.239729 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:36.239820 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":251,"threads_started":1}
I20260924 03:12:36.336333 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:36.335432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:43823)
0924 03:12:36.336054 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:36.336141 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":255,"threads_started":1}
I20260924 03:12:36.474308 15846 raft_consensus.cc:482] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:12:36.474563 15846 raft_consensus.cc:504] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } } peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } }
I20260924 03:12:36.475275 15846 leader_election.cc:288] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42041e2066ab4a6d878e1e2a374d5301 (127.0.97.2:33695), 78f63d155fdc4959869cf5134eb15498 (127.0.97.3:38731)
W20260924 03:12:36.476307 15457 leader_election.cc:341] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42041e2066ab4a6d878e1e2a374d5301 (127.0.97.2:33695): Not found: Tablet not found: 2bf2d96397dd4e9281ae54958ded53fd
W20260924 03:12:36.476495 15455 leader_election.cc:341] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 78f63d155fdc4959869cf5134eb15498 (127.0.97.3:38731): Not found: Tablet not found: 2bf2d96397dd4e9281ae54958ded53fd
I20260924 03:12:36.476641 15455 leader_election.cc:302] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a953cfe71cb49dda86b666b75961cc2; no voters: 42041e2066ab4a6d878e1e2a374d5301, 78f63d155fdc4959869cf5134eb15498
I20260924 03:12:36.476933 15846 raft_consensus.cc:2726] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:12:36.786438 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:36.785762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.2:40727)
0924 03:12:36.786201 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:36.786276 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":234,"threads_started":1}
I20260924 03:12:37.241882 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:37.241083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.3:43581)
0924 03:12:37.241599 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:37.241697 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":276,"threads_started":1}
I20260924 03:12:37.338315 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:37.337592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:53315)
0924 03:12:37.338070 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:37.338155 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":240,"threads_started":1}
I20260924 03:12:37.788141 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:37.787386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.2:47171)
0924 03:12:37.787838 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:37.787936 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":232,"threads_started":1}
I20260924 03:12:38.243758 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:38.242898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.3:40807)
0924 03:12:38.243498 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:38.243590 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":305,"threads_started":1}
I20260924 03:12:38.340267 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:38.339540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:47461)
0924 03:12:38.340006 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:38.340089 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":236,"threads_started":1}
I20260924 03:12:38.789927 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:38.789174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.2:59439)
0924 03:12:38.789687 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:38.789768 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":254,"threads_started":1}
I20260924 03:12:39.245735 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:39.244911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.3:53371)
0924 03:12:39.245469 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:39.245573 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":246,"threads_started":1}
I20260924 03:12:39.342016 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:39.341313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:52663)
0924 03:12:39.341767 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:39.341847 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":267,"threads_started":1}
I20260924 03:12:39.778713 15856 raft_consensus.cc:482] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:12:39.778923 15856 raft_consensus.cc:504] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } } peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } }
I20260924 03:12:39.779735 15856 leader_election.cc:288] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42041e2066ab4a6d878e1e2a374d5301 (127.0.97.2:33695), 78f63d155fdc4959869cf5134eb15498 (127.0.97.3:38731)
W20260924 03:12:39.780727 15457 leader_election.cc:341] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42041e2066ab4a6d878e1e2a374d5301 (127.0.97.2:33695): Not found: Tablet not found: 2bf2d96397dd4e9281ae54958ded53fd
W20260924 03:12:39.780994 15455 leader_election.cc:341] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 78f63d155fdc4959869cf5134eb15498 (127.0.97.3:38731): Not found: Tablet not found: 2bf2d96397dd4e9281ae54958ded53fd
I20260924 03:12:39.781168 15455 leader_election.cc:302] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a953cfe71cb49dda86b666b75961cc2; no voters: 42041e2066ab4a6d878e1e2a374d5301, 78f63d155fdc4959869cf5134eb15498
I20260924 03:12:39.781486 15856 raft_consensus.cc:2726] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:12:39.791702 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:39.790903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.2:52115)
0924 03:12:39.791446 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:39.791527 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":320,"threads_started":1}
I20260924 03:12:40.247716 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.246748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.3:43645)
0924 03:12:40.247312 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.247504 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":319,"threads_started":1}
I20260924 03:12:40.344137 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.343179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:57195)
0924 03:12:40.343817 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.343935 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":418,"threads_started":1}
I20260924 03:12:40.793857 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.792902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.2:51941)
0924 03:12:40.793486 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.793670 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":276,"threads_started":1}
I20260924 03:12:41.250231 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:41.249082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.3:37569)
0924 03:12:41.249859 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:41.250018 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":511,"threads_started":1}
I20260924 03:12:41.346065 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:41.345195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:51983)
0924 03:12:41.345754 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:41.345852 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":344,"threads_started":1}
I20260924 03:12:41.795923 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:41.795168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.2:37021)
0924 03:12:41.795647 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:41.795735 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":274,"threads_started":1}
I20260924 03:12:42.252067 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:42.251348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.3:39147)
0924 03:12:42.251789 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:42.251873 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":233,"threads_started":1}
I20260924 03:12:42.348016 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:42.347199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:60627)
0924 03:12:42.347721 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:42.347818 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":231,"threads_started":1}
I20260924 03:12:42.646209 15866 raft_consensus.cc:482] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:12:42.646421 15866 raft_consensus.cc:504] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } } peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } }
I20260924 03:12:42.647217 15866 leader_election.cc:288] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42041e2066ab4a6d878e1e2a374d5301 (127.0.97.2:33695), 78f63d155fdc4959869cf5134eb15498 (127.0.97.3:38731)
W20260924 03:12:42.648255 15455 leader_election.cc:341] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 78f63d155fdc4959869cf5134eb15498 (127.0.97.3:38731): Not found: Tablet not found: 2bf2d96397dd4e9281ae54958ded53fd
W20260924 03:12:42.648466 15457 leader_election.cc:341] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42041e2066ab4a6d878e1e2a374d5301 (127.0.97.2:33695): Not found: Tablet not found: 2bf2d96397dd4e9281ae54958ded53fd
I20260924 03:12:42.648715 15457 leader_election.cc:302] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a953cfe71cb49dda86b666b75961cc2; no voters: 42041e2066ab4a6d878e1e2a374d5301, 78f63d155fdc4959869cf5134eb15498
I20260924 03:12:42.649034 15866 raft_consensus.cc:2726] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:12:42.798069 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:42.797227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.2:49439)
0924 03:12:42.797794 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:42.797897 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":256,"threads_started":1}
I20260924 03:12:43.254132 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.253143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.3:40561)
0924 03:12:43.253839 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.253958 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":343,"threads_started":1}
I20260924 03:12:43.350255 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.349394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:56659)
0924 03:12:43.349973 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.350083 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":343,"threads_started":1}
I20260924 03:12:43.800256 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.799181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.2:41225)
0924 03:12:43.799940 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.800028 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":430,"threads_started":1}
I20260924 03:12:44.255927 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:44.255153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.3:47841)
0924 03:12:44.255650 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:44.255735 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":235,"threads_started":1}
I20260924 03:12:44.352149 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:44.351344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:38833)
0924 03:12:44.351830 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:44.351915 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":259,"threads_started":1}
I20260924 03:12:44.802225 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:44.801405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.2:60641)
0924 03:12:44.801947 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:44.802056 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":277,"threads_started":1}
I20260924 03:12:45.258091 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:45.257122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.3:49301)
0924 03:12:45.257753 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:45.257873 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":258,"threads_started":1}
I20260924 03:12:45.354131 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:45.353308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:40231)
0924 03:12:45.353828 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:45.353919 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":310,"threads_started":1}
I20260924 03:12:45.804251 15876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:45.803416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.2:39545)
0924 03:12:45.803985 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:45.804073 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":340,"threads_started":1}
I20260924 03:12:46.010635 15877 raft_consensus.cc:482] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:12:46.010824 15877 raft_consensus.cc:504] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42041e2066ab4a6d878e1e2a374d5301" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "78f63d155fdc4959869cf5134eb15498" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38731 } } peers { permanent_uuid: "2a953cfe71cb49dda86b666b75961cc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45013 } }
I20260924 03:12:46.011514 15877 leader_election.cc:288] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42041e2066ab4a6d878e1e2a374d5301 (127.0.97.2:33695), 78f63d155fdc4959869cf5134eb15498 (127.0.97.3:38731)
W20260924 03:12:46.012676 15455 leader_election.cc:341] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 78f63d155fdc4959869cf5134eb15498 (127.0.97.3:38731): Not found: Tablet not found: 2bf2d96397dd4e9281ae54958ded53fd
W20260924 03:12:46.012847 15457 leader_election.cc:341] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42041e2066ab4a6d878e1e2a374d5301 (127.0.97.2:33695): Not found: Tablet not found: 2bf2d96397dd4e9281ae54958ded53fd
I20260924 03:12:46.013057 15457 leader_election.cc:302] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a953cfe71cb49dda86b666b75961cc2; no voters: 42041e2066ab4a6d878e1e2a374d5301, 78f63d155fdc4959869cf5134eb15498
I20260924 03:12:46.013339 15877 raft_consensus.cc:2726] T 2bf2d96397dd4e9281ae54958ded53fd P 2a953cfe71cb49dda86b666b75961cc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:12:46.260253 15878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:46.259233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.3:33713)
0924 03:12:46.259953 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:46.260059 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":477,"threads_started":1}
I20260924 03:12:46.356287 15879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:46.355313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:46225)
0924 03:12:46.355927 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:46.356075 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":324,"threads_started":1}
I20260924 03:12:46.806648 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:46.805362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.2:58587)
0924 03:12:46.806038 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:46.806193 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":304,"threads_started":1}
I20260924 03:12:47.262171 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:47.261286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.3:50305)
0924 03:12:47.261846 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:47.261979 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":311,"threads_started":1}
I20260924 03:12:47.358397 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:47.357653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:41883)
0924 03:12:47.358125 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:47.358211 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":252,"threads_started":1}
I20260924 03:12:47.809008 15883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:47.807879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.2:36379)
0924 03:12:47.808481 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:47.808782 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":304,"threads_started":1}
I20260924 03:12:48.264325 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:48.263277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.3:34065)
0924 03:12:48.263952 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:48.264049 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":431,"threads_started":1}
I20260924 03:12:48.360818 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:48.359822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:54373)
0924 03:12:48.360443 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:48.360614 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":369,"threads_started":1}
I20260924 03:12:48.811295 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:48.810530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.2:49485)
0924 03:12:48.811036 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:48.811127 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":280,"threads_started":1}
I20260924 03:12:49.237504 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 15404
I20260924 03:12:49.254433 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:49.238760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36456 (local address 127.0.97.1:45013)
0924 03:12:49.239334 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:49.239344 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:49.239377 (+ 33us) server_negotiation.cc:407] Connection header received
0924 03:12:49.239475 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:49.239490 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:49.239807 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:49.239996 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:49.241653 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:49.242734 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:49.244212 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:49.245097 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:49.245205 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:49.248002 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:49.248016 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:49.248036 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:49.248080 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:49.251298 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:49.252028 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:49.252035 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:49.252037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:49.252176 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:49.252651 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:49.252654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:49.252656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:49.253022 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:49.253255 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:49.253793 (+ 538us) server_negotiation.cc:299] Negotiation successful
0924 03:12:49.254141 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":239,"threads_started":1}
I20260924 03:12:49.255726 15563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:49.266564 15889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:49.265574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.3:38265)
0924 03:12:49.266171 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:49.266299 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":284,"threads_started":1}
I20260924 03:12:49.354303 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 15404
I20260924 03:12:49.374429 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 15572
I20260924 03:12:49.389966 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:49.375296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42010 (local address 127.0.97.2:33695)
0924 03:12:49.375843 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:49.375848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:49.375863 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:12:49.375917 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:49.375921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:49.375978 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:49.376108 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:49.377592 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:49.378390 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:49.379663 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:49.380080 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:49.380220 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:49.383322 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:49.383328 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:49.383331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:49.383368 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:49.386848 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:49.387702 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:49.387710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:49.387718 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:49.387863 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:49.388285 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:49.388294 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:49.388296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:49.388684 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:49.388959 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:49.389305 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 03:12:49.389587 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":335,"threads_started":1}
I20260924 03:12:49.390847 15695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:49.478364 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 15572
I20260924 03:12:49.496662 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 15703
I20260924 03:12:49.511212 15892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:49.497527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40746 (local address 127.0.97.3:38731)
0924 03:12:49.498154 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:49.498158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:49.498175 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:12:49.498229 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:49.498233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:49.498301 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:49.498395 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:49.499764 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:49.500700 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:49.502026 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:49.502432 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:49.502529 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:49.505418 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:49.505423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:49.505426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:49.505460 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:49.508307 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:49.509087 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:49.509091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:49.509093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:49.509238 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:49.509584 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:49.509588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:49.509599 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:49.509881 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:49.510118 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:49.510461 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 03:12:49.510693 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":343,"threads_started":1}
I20260924 03:12:49.511919 15826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:49.596673 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 15703
2026-09-24T03:12:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24890 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:49 dist-test-slave-5l98 krb5kdc[15896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:49 dist-test-slave-5l98 krb5kdc[15896](info): set up 2 sockets
Sep 24 03:12:49 dist-test-slave-5l98 krb5kdc[15896](info): commencing operation
krb5kdc: starting...
W20260924 03:12:50.749285 388 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:12:50 dist-test-slave-5l98 krb5kdc[15896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:12:50.997090 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32789
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:51.330268 15909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:51.330575 15909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:51.330715 15909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:51.337983 15909 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 03:12:51.338086 15909 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 03:12:51.338168 15909 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 03:12:51.338246 15909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:51.338354 15909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:51.338472 15909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:51.338568 15909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:51.338647 15909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:51.349040 15909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:51.350417 15909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:51.351789 15909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:12:51.364892 15909 server_base.cc:1034] running on GCE node
W20260924 03:12:51.364923 15915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:51.364986 15917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:51.364892 15914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:51.366287 15909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:51.367906 15909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:51.369196 15909 hybrid_clock.cc:648] HybridClock initialized: now 1790219571369126 us; error 33 us; skew 500 ppm
I20260924 03:12:51.369621 15909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:51 dist-test-slave-5l98 krb5kdc[15896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:51.377317 15909 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:12:51.378901 15909 webserver.cc:460] Webserver started at http://127.0.97.62:33231/ using document root <none> and password file <none>
I20260924 03:12:51.379588 15909 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:51.379678 15909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:51.379940 15909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:51.383366 15909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "369d40362da64878af413665ae5a3184"
format_stamp: "Formatted at 2026-09-24 03:12:51 on dist-test-slave-5l98"
server_key: "8226170246af57e1628c4e9bbfa0cdcf"
server_key_iv: "2af8407c6c7bf93e30d0cf3f258b7b28"
server_key_version: "encryptionkey@0"
I20260924 03:12:51.384194 15909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "369d40362da64878af413665ae5a3184"
format_stamp: "Formatted at 2026-09-24 03:12:51 on dist-test-slave-5l98"
server_key: "8226170246af57e1628c4e9bbfa0cdcf"
server_key_iv: "2af8407c6c7bf93e30d0cf3f258b7b28"
server_key_version: "encryptionkey@0"
I20260924 03:12:51.389581 15909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 03:12:51.393200 15924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:51.394595 15909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:12:51.394876 15909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "369d40362da64878af413665ae5a3184"
format_stamp: "Formatted at 2026-09-24 03:12:51 on dist-test-slave-5l98"
server_key: "8226170246af57e1628c4e9bbfa0cdcf"
server_key_iv: "2af8407c6c7bf93e30d0cf3f258b7b28"
server_key_version: "encryptionkey@0"
I20260924 03:12:51.395126 15909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:51.411134 15909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:51.415934 15909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:51.416246 15909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:51.436005 15909 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32789
I20260924 03:12:51.436018 15975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32789 every 8 connection(s)
I20260924 03:12:51.437377 15909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:51.441247 15976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:51.441623 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 15909
I20260924 03:12:51.441833 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:12:51.442515 388 external_mini_cluster.cc:1428] Setting key a80c3d286c857dcb48a664b1958ae7e5
I20260924 03:12:51.456095 15976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184: Bootstrap starting.
Sep 24 03:12:51 dist-test-slave-5l98 krb5kdc[15896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:12:51.459345 15976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:51.460913 15976 log.cc:824] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:51.464354 15976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184: No bootstrap required, opened a new log
I20260924 03:12:51.468391 15979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:51.444567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48604 (local address 127.0.97.62:32789)
0924 03:12:51.445380 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:51.445396 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:51.445443 (+ 47us) server_negotiation.cc:407] Connection header received
0924 03:12:51.448084 (+ 2641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:51.448114 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:51.448646 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:51.449116 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:51.450446 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:51.452450 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:51.453807 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:51.454843 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:51.455038 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:51.459369 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:51.459382 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:51.459403 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:51.459465 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:51.463500 (+ 4035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:51.464289 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:51.464297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:51.464306 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:51.464571 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:51.464984 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:51.464991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:51.464997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:51.465724 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:51.466043 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:51.466437 (+ 394us) server_negotiation.cc:299] Negotiation successful
0924 03:12:51.466917 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":333,"threads_started":1}
I20260924 03:12:51.474328 15976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369d40362da64878af413665ae5a3184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32789 } }
I20260924 03:12:51.474715 15976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:51.474781 15976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 369d40362da64878af413665ae5a3184, State: Initialized, Role: FOLLOWER
I20260924 03:12:51.475286 15976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369d40362da64878af413665ae5a3184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32789 } }
I20260924 03:12:51.475474 15976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:51.475572 15976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:51.475693 15976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:51.478281 15976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369d40362da64878af413665ae5a3184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32789 } }
I20260924 03:12:51.478730 15976 leader_election.cc:302] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 369d40362da64878af413665ae5a3184; no voters:
I20260924 03:12:51.479281 15976 leader_election.cc:288] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:51.479450 15981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:51.480103 15981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 1 LEADER]: Becoming Leader. State: Replica: 369d40362da64878af413665ae5a3184, State: Running, Role: LEADER
I20260924 03:12:51.480783 15981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369d40362da64878af413665ae5a3184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32789 } }
I20260924 03:12:51.481415 15976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:51.484622 15982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "369d40362da64878af413665ae5a3184" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369d40362da64878af413665ae5a3184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32789 } } }
I20260924 03:12:51.484566 15983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 369d40362da64878af413665ae5a3184. Latest consensus state: current_term: 1 leader_uuid: "369d40362da64878af413665ae5a3184" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369d40362da64878af413665ae5a3184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32789 } } }
I20260924 03:12:51.485078 15983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:51.485055 15982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:51.486699 15990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:51.492460 15990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:51.501019 15990 catalog_manager.cc:1347] Generated new cluster ID: 08a9c020672f496cb5d322c663646fee
I20260924 03:12:51.501144 15990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:51.512110 15990 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:51.514038 15990 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:51.523566 15990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184: Generated new TSK 0
I20260924 03:12:51.524478 15990 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 03:12:51.596410 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32789
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:51.935269 16002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:51.935560 16002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:51.935736 16002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:51.943019 16002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:51.943135 16002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:51.943315 16002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:12:51.953488 16002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:51.954823 16002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:51.956139 16002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:51.969527 16007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:51.969527 16008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:51.969812 16010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:51.970433 16002 server_base.cc:1034] running on GCE node
I20260924 03:12:51.971047 16002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:51.972114 16002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:51.973326 16002 hybrid_clock.cc:648] HybridClock initialized: now 1790219571973268 us; error 49 us; skew 500 ppm
I20260924 03:12:51.973766 16002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:51 dist-test-slave-5l98 krb5kdc[15896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:51.977177 16002 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:12:51.978822 16002 webserver.cc:460] Webserver started at http://127.0.97.1:45865/ using document root <none> and password file <none>
I20260924 03:12:51.979499 16002 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:51.979658 16002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:51.979974 16002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:51.983143 16002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "cb30ddfc6df9492e8b6223d6e9ccd912"
format_stamp: "Formatted at 2026-09-24 03:12:51 on dist-test-slave-5l98"
server_key: "e50e433e8125fcb44e63e74c921a8eeb"
server_key_iv: "39a56c65beed25b89c173e247772a434"
server_key_version: "encryptionkey@0"
I20260924 03:12:51.983908 16002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cb30ddfc6df9492e8b6223d6e9ccd912"
format_stamp: "Formatted at 2026-09-24 03:12:51 on dist-test-slave-5l98"
server_key: "e50e433e8125fcb44e63e74c921a8eeb"
server_key_iv: "39a56c65beed25b89c173e247772a434"
server_key_version: "encryptionkey@0"
I20260924 03:12:51.989212 16002 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 03:12:51.992705 16017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:51.993969 16002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:12:51.994141 16002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "cb30ddfc6df9492e8b6223d6e9ccd912"
format_stamp: "Formatted at 2026-09-24 03:12:51 on dist-test-slave-5l98"
server_key: "e50e433e8125fcb44e63e74c921a8eeb"
server_key_iv: "39a56c65beed25b89c173e247772a434"
server_key_version: "encryptionkey@0"
I20260924 03:12:51.994375 16002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:52.007884 16002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:52.012213 16002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:52.012669 16002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:52.013705 16002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:52.015143 16002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:52.015254 16002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:52.015410 16002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:52.015501 16002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:52.048713 16002 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42661
I20260924 03:12:52.048758 16129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42661 every 8 connection(s)
I20260924 03:12:52.050052 16002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:12:52.052354 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 16002
I20260924 03:12:52.052654 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:12:52.053476 388 external_mini_cluster.cc:1428] Setting key cf246914ab0fd69e6449cd66b830a4c1
Sep 24 03:12:52 dist-test-slave-5l98 krb5kdc[15896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:12:52.072599 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:52.052815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42593 (local address 127.0.97.62:32789)
0924 03:12:52.053338 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:52.053347 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:52.054399 (+ 1052us) server_negotiation.cc:407] Connection header received
0924 03:12:52.056287 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:52.056294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:52.056387 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:52.056537 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:52.058171 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:52.059348 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:52.061142 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:52.061856 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:52.062006 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:52.066461 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:52.066466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:52.066469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:52.066504 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:52.069361 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:52.070172 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:52.070181 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:52.070184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:52.070305 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:52.070760 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:52.070770 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:52.070772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:52.071044 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:52.071207 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:52.071947 (+ 740us) server_negotiation.cc:299] Negotiation successful
0924 03:12:52.072187 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":238,"threads_started":1}
I20260924 03:12:52.073730 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:52.053375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.1:42593)
0924 03:12:52.054230 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:52.054284 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:52.056005 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:52.056832 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:52.056840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:52.057380 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:52.057955 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:52.057972 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:52.059542 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:52.059546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:52.060987 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:52.061000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:52.062075 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:52.062079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:52.062325 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:52.063276 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:52.063313 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:52.066180 (+ 2867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:52.069539 (+ 3359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:52.069550 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:52.069569 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:52.070013 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:52.070423 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:52.070428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:52.070432 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:52.070633 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:52.071179 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:52.071187 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:52.071554 (+ 367us) client_negotiation.cc:769] Sending connection context
0924 03:12:52.071835 (+ 281us) client_negotiation.cc:240] Negotiation successful
0924 03:12:52.072415 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":391,"threads_started":1}
I20260924 03:12:52.075555 16130 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32789
I20260924 03:12:52.075933 16130 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:52.076898 16130 heartbeater.cc:511] Master 127.0.97.62:32789 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 03:12:52.079278 15941 ts_manager.cc:194] Registered new tserver with Master: cb30ddfc6df9492e8b6223d6e9ccd912 (127.0.97.1:42661)
I20260924 03:12:52.080775 15941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 03:12:52.113039 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32789
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:52.444880 16136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:52.445211 16136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:52.445381 16136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:52.452844 16136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:52.452993 16136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:52.453182 16136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:12:52.463245 16136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:52.464684 16136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:52.465978 16136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:52.479727 16141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:52.479776 16142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:52.479683 16144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:52.480890 16136 server_base.cc:1034] running on GCE node
I20260924 03:12:52.481580 16136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:52.482688 16136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:52.483909 16136 hybrid_clock.cc:648] HybridClock initialized: now 1790219572483849 us; error 69 us; skew 500 ppm
I20260924 03:12:52.484251 16136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:52 dist-test-slave-5l98 krb5kdc[15896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:52.487728 16136 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:12:52.489343 16136 webserver.cc:460] Webserver started at http://127.0.97.2:43479/ using document root <none> and password file <none>
I20260924 03:12:52.490056 16136 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:52.490200 16136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:52.490518 16136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:52.493526 16136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "9b765b6326824826a0d2e92c7859ac00"
format_stamp: "Formatted at 2026-09-24 03:12:52 on dist-test-slave-5l98"
server_key: "9a54bed913ba8bc2dbf23a8945558336"
server_key_iv: "864c70f9bdc29919e6f2a3f047da0643"
server_key_version: "encryptionkey@0"
I20260924 03:12:52.494269 16136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9b765b6326824826a0d2e92c7859ac00"
format_stamp: "Formatted at 2026-09-24 03:12:52 on dist-test-slave-5l98"
server_key: "9a54bed913ba8bc2dbf23a8945558336"
server_key_iv: "864c70f9bdc29919e6f2a3f047da0643"
server_key_version: "encryptionkey@0"
I20260924 03:12:52.499912 16136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:12:52.503695 16151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:52.505118 16136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 03:12:52.505304 16136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "9b765b6326824826a0d2e92c7859ac00"
format_stamp: "Formatted at 2026-09-24 03:12:52 on dist-test-slave-5l98"
server_key: "9a54bed913ba8bc2dbf23a8945558336"
server_key_iv: "864c70f9bdc29919e6f2a3f047da0643"
server_key_version: "encryptionkey@0"
I20260924 03:12:52.505509 16136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:52.518980 16136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:52.523522 16136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:52.523841 16136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:52.524946 16136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:52.526274 16136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:52.526382 16136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:52.526535 16136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:52.526611 16136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:52.562573 16136 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40015
I20260924 03:12:52.562638 16263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40015 every 8 connection(s)
I20260924 03:12:52.563881 16136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:12:52.566314 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 16136
I20260924 03:12:52.566507 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:12:52.567042 388 external_mini_cluster.cc:1428] Setting key b07e94f33990a1e8f1d810a36f7fa91c
Sep 24 03:12:52 dist-test-slave-5l98 krb5kdc[15896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:12:52.586455 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:52.566785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47319 (local address 127.0.97.62:32789)
0924 03:12:52.567008 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:52.567014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:52.568278 (+ 1264us) server_negotiation.cc:407] Connection header received
0924 03:12:52.570214 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:52.570223 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:52.570316 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:52.570427 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:52.571920 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:52.573090 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:52.574633 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:52.575379 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:52.575522 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:52.579079 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:52.579088 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:52.579096 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:52.579142 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:52.582841 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:52.583673 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:52.583682 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:52.583688 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:52.583860 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:52.584310 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:52.584317 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:52.584323 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:52.584733 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:52.584888 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:52.585776 (+ 888us) server_negotiation.cc:299] Negotiation successful
0924 03:12:52.586108 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:12:52.587532 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:52.567281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.2:47319)
0924 03:12:52.568108 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:52.568156 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:52.569900 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:52.570647 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:52.570655 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:52.571183 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:52.571745 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:52.571762 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:52.573254 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:52.573258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:52.574459 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:52.574470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:52.575601 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:52.575606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:52.575803 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:52.576575 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:52.576605 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:52.578818 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:52.583068 (+ 4250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:52.583076 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:52.583090 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:52.583539 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:52.583988 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:52.583992 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:52.583994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:52.584191 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:52.584938 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:52.584945 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:52.585295 (+ 350us) client_negotiation.cc:769] Sending connection context
0924 03:12:52.585692 (+ 397us) client_negotiation.cc:240] Negotiation successful
0924 03:12:52.586256 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":372,"threads_started":1}
I20260924 03:12:52.589147 16264 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32789
I20260924 03:12:52.589514 16264 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:52.590432 16264 heartbeater.cc:511] Master 127.0.97.62:32789 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 03:12:52.592327 15941 ts_manager.cc:194] Registered new tserver with Master: 9b765b6326824826a0d2e92c7859ac00 (127.0.97.2:40015)
I20260924 03:12:52.593263 15941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 03:12:52.625538 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32789
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:52.950995 16269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:52.951280 16269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:52.951375 16269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:52.958783 16269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:52.958894 16269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:52.959012 16269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:12:52.968961 16269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:12:52.970234 16269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:52.971565 16269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:52.984754 16275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:52.984781 16277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:52.984715 16274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:52.985661 16269 server_base.cc:1034] running on GCE node
I20260924 03:12:52.986275 16269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:52.987295 16269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:52.988694 16269 hybrid_clock.cc:648] HybridClock initialized: now 1790219572988632 us; error 49 us; skew 500 ppm
I20260924 03:12:52.989030 16269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:12:52 dist-test-slave-5l98 krb5kdc[15896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:52.992345 16269 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:12:52.993988 16269 webserver.cc:460] Webserver started at http://127.0.97.3:45603/ using document root <none> and password file <none>
I20260924 03:12:52.994674 16269 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:52.994822 16269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:52.995131 16269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:52.998215 16269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "7371cf39fb024460af08c87a3e1269d9"
format_stamp: "Formatted at 2026-09-24 03:12:52 on dist-test-slave-5l98"
server_key: "d8337a2a1868243069a8139c031e4ad0"
server_key_iv: "fa40babc994cf5fda7c4d13f1ccc3593"
server_key_version: "encryptionkey@0"
I20260924 03:12:52.999095 16269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7371cf39fb024460af08c87a3e1269d9"
format_stamp: "Formatted at 2026-09-24 03:12:52 on dist-test-slave-5l98"
server_key: "d8337a2a1868243069a8139c031e4ad0"
server_key_iv: "fa40babc994cf5fda7c4d13f1ccc3593"
server_key_version: "encryptionkey@0"
I20260924 03:12:53.004344 16269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:12:53.008046 16284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:53.009583 16269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:12:53.009778 16269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "7371cf39fb024460af08c87a3e1269d9"
format_stamp: "Formatted at 2026-09-24 03:12:52 on dist-test-slave-5l98"
server_key: "d8337a2a1868243069a8139c031e4ad0"
server_key_iv: "fa40babc994cf5fda7c4d13f1ccc3593"
server_key_version: "encryptionkey@0"
I20260924 03:12:53.009949 16269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:53.030404 16269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:53.035038 16269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:53.035272 16269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:53.036185 16269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:53.037530 16269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:53.037599 16269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:53.037683 16269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:53.037725 16269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:53.071882 16269 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38127
I20260924 03:12:53.071923 16396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38127 every 8 connection(s)
I20260924 03:12:53.073405 16269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:12:53.079145 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 16269
I20260924 03:12:53.079344 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:12:53.080024 388 external_mini_cluster.cc:1428] Setting key f219500032420e1a438239b6293460fa
I20260924 03:12:53.083935 16130 heartbeater.cc:503] Master 127.0.97.62:32789 was elected leader, sending a full tablet report...
Sep 24 03:12:53 dist-test-slave-5l98 krb5kdc[15896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:12:53.095182 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:53.076222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45897 (local address 127.0.97.62:32789)
0924 03:12:53.076450 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:53.076457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:53.077584 (+ 1127us) server_negotiation.cc:407] Connection header received
0924 03:12:53.079719 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:53.079724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:53.079794 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:53.079876 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:53.081981 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.083140 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.084867 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.085491 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.085635 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:53.089344 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:53.089350 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:53.089353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:53.089385 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:53.092042 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:53.092860 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:53.092872 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:53.092875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:53.093010 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:53.093452 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:53.093463 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:53.093465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:53.093735 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:53.093898 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:53.094581 (+ 683us) server_negotiation.cc:299] Negotiation successful
0924 03:12:53.094827 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 03:12:53.096251 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:53.076671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.3:45897)
0924 03:12:53.077405 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:53.077453 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:53.079401 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:53.080166 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:53.080179 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:53.080919 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:53.081810 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:53.081836 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.083293 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.083299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:53.084692 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:53.084704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.085793 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.085802 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:53.086175 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:53.086817 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:53.086843 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:53.089089 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:53.092198 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:53.092208 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:53.092226 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:53.092731 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:53.093136 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:53.093141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:53.093145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:53.093330 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:53.093869 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:53.093877 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:53.094209 (+ 332us) client_negotiation.cc:769] Sending connection context
0924 03:12:53.094471 (+ 262us) client_negotiation.cc:240] Negotiation successful
0924 03:12:53.094997 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":297,"threads_started":1}
I20260924 03:12:53.097673 16397 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32789
I20260924 03:12:53.098112 16397 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:53.098927 16397 heartbeater.cc:511] Master 127.0.97.62:32789 requested a full tablet report, sending...
I20260924 03:12:53.100603 15941 ts_manager.cc:194] Registered new tserver with Master: 7371cf39fb024460af08c87a3e1269d9 (127.0.97.3:38127)
I20260924 03:12:53.101472 15941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45897
I20260924 03:12:53.107689 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:12:53.124946 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:53.112298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48610 (local address 127.0.97.62:32789)
0924 03:12:53.112466 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:53.112471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:53.113026 (+ 555us) server_negotiation.cc:407] Connection header received
0924 03:12:53.113092 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:53.113097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:53.113161 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:53.113328 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:53.114846 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.115645 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.117061 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.117468 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.117566 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:53.119872 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:53.119877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:53.119880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:53.119910 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:53.122485 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:53.123157 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:53.123161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:53.123163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:53.123276 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:53.123677 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:53.123681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:53.123683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:53.123995 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:53.124170 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:53.124421 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 03:12:53.124729 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:12:53.129042 15941 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:12:53.135984 15941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:12:53.172047 16413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:53.148301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40015 (local address 127.0.0.1:60398)
0924 03:12:53.157459 (+ 9158us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:53.157539 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:53.157822 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:53.163804 (+ 5982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:53.163812 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:53.163853 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:53.164371 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:53.164388 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.166403 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.166410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:53.169072 (+ 2662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:53.169097 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.170898 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.170905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:53.171145 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:53.171227 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 03:12:53.171347 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 03:12:53.171559 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8772,"mutex_wait_us":52,"thread_start_us":15064,"threads_started":1}
I20260924 03:12:53.172374 16410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:53.148375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.0.97.2:40015)
0924 03:12:53.162741 (+ 14366us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:53.162752 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:53.162788 (+ 36us) server_negotiation.cc:407] Connection header received
0924 03:12:53.162912 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:53.162925 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:53.163243 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:53.163466 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:53.164647 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.166253 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.169258 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.170682 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.170817 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:53.171019 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:53.171818 (+ 799us) server_negotiation.cc:299] Negotiation successful
0924 03:12:53.172013 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14202,"thread_start_us":14079,"threads_started":1}
I20260924 03:12:53.172813 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:53.148476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42661 (local address 127.0.0.1:37798)
0924 03:12:53.163055 (+ 14579us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:53.163075 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:53.163220 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:53.164702 (+ 1482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:53.164709 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:53.164742 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:53.165157 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:53.165171 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.167230 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.167238 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:53.169550 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:53.169566 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.171153 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.171159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:53.172069 (+ 910us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:53.172139 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 03:12:53.172233 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 03:12:53.172347 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14435,"mutex_wait_us":203,"thread_start_us":15689,"threads_started":1}
I20260924 03:12:53.173097 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:53.148768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37798 (local address 127.0.97.1:42661)
0924 03:12:53.164072 (+ 15304us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:53.164080 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:53.164106 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:12:53.164217 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:53.164224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:53.164466 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:53.165387 (+ 921us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:53.165962 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.167083 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.169779 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.170917 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.171222 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:53.171687 (+ 465us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:53.172630 (+ 943us) server_negotiation.cc:299] Negotiation successful
0924 03:12:53.172795 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15119,"thread_start_us":14954,"threads_started":1}
I20260924 03:12:53.173789 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:53.150573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32798 (local address 127.0.97.3:38127)
0924 03:12:53.158641 (+ 8068us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:53.158654 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:53.164689 (+ 6035us) server_negotiation.cc:407] Connection header received
0924 03:12:53.165012 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:53.165019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:53.165236 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:53.165396 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:53.166143 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.167742 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.170104 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.171427 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.171548 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:53.171768 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:53.173092 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0924 03:12:53.173396 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7823,"thread_start_us":7627,"threads_started":1}
I20260924 03:12:53.174276 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:53.149895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38127 (local address 127.0.0.1:32798)
0924 03:12:53.164542 (+ 14647us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:53.164560 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:53.164809 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:53.165645 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:53.165652 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:53.165678 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:53.165965 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:53.165972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.167912 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.167918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:53.169940 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:53.169954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.171940 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.171947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:53.172410 (+ 463us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:53.172620 (+ 210us) client_negotiation.cc:769] Sending connection context
0924 03:12:53.172725 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 03:12:53.173950 (+ 1225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14458,"mutex_wait_us":317,"thread_start_us":6997,"threads_started":1}
I20260924 03:12:53.176007 16199 tablet_service.cc:1467] Processing CreateTablet for tablet c3a0ac2f5ddf4fbeb9d11eb48fa17b8f (DEFAULT_TABLE table=test-table [id=793dd46a4f604a3aa6800db98253060f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:53.177528 16199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3a0ac2f5ddf4fbeb9d11eb48fa17b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:53.177196 16065 tablet_service.cc:1467] Processing CreateTablet for tablet c3a0ac2f5ddf4fbeb9d11eb48fa17b8f (DEFAULT_TABLE table=test-table [id=793dd46a4f604a3aa6800db98253060f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:53.178009 16332 tablet_service.cc:1467] Processing CreateTablet for tablet c3a0ac2f5ddf4fbeb9d11eb48fa17b8f (DEFAULT_TABLE table=test-table [id=793dd46a4f604a3aa6800db98253060f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:53.179375 16065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3a0ac2f5ddf4fbeb9d11eb48fa17b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:53.179374 16332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3a0ac2f5ddf4fbeb9d11eb48fa17b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:53.191644 16415 tablet_bootstrap.cc:492] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00: Bootstrap starting.
I20260924 03:12:53.194025 16417 tablet_bootstrap.cc:492] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9: Bootstrap starting.
I20260924 03:12:53.196097 16415 tablet_bootstrap.cc:654] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:53.197157 16416 tablet_bootstrap.cc:492] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912: Bootstrap starting.
I20260924 03:12:53.197845 16415 log.cc:824] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:53.198800 16417 tablet_bootstrap.cc:654] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:53.199944 16417 log.cc:824] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:53.200764 16416 tablet_bootstrap.cc:654] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:53.201462 16415 tablet_bootstrap.cc:492] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00: No bootstrap required, opened a new log
I20260924 03:12:53.201740 16415 ts_tablet_manager.cc:1397] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 03:12:53.201922 16416 log.cc:824] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:53.202701 16417 tablet_bootstrap.cc:492] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9: No bootstrap required, opened a new log
I20260924 03:12:53.202970 16417 ts_tablet_manager.cc:1397] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20260924 03:12:53.204818 16416 tablet_bootstrap.cc:492] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912: No bootstrap required, opened a new log
I20260924 03:12:53.205072 16416 ts_tablet_manager.cc:1397] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 03:12:53.210945 16417 raft_consensus.cc:348] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b765b6326824826a0d2e92c7859ac00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40015 } } peers { permanent_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42661 } } peers { permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 } }
I20260924 03:12:53.211390 16417 raft_consensus.cc:374] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:53.211493 16417 raft_consensus.cc:729] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7371cf39fb024460af08c87a3e1269d9, State: Initialized, Role: FOLLOWER
I20260924 03:12:53.212296 16417 consensus_queue.cc:260] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b765b6326824826a0d2e92c7859ac00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40015 } } peers { permanent_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42661 } } peers { permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 } }
I20260924 03:12:53.212932 16415 raft_consensus.cc:348] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b765b6326824826a0d2e92c7859ac00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40015 } } peers { permanent_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42661 } } peers { permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 } }
I20260924 03:12:53.213344 16415 raft_consensus.cc:374] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:53.213428 16415 raft_consensus.cc:729] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b765b6326824826a0d2e92c7859ac00, State: Initialized, Role: FOLLOWER
I20260924 03:12:53.214052 16415 consensus_queue.cc:260] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b765b6326824826a0d2e92c7859ac00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40015 } } peers { permanent_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42661 } } peers { permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 } }
I20260924 03:12:53.215481 16264 heartbeater.cc:503] Master 127.0.97.62:32789 was elected leader, sending a full tablet report...
I20260924 03:12:53.215693 16416 raft_consensus.cc:348] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b765b6326824826a0d2e92c7859ac00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40015 } } peers { permanent_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42661 } } peers { permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 } }
I20260924 03:12:53.216003 16415 ts_tablet_manager.cc:1428] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260924 03:12:53.216128 16416 raft_consensus.cc:374] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:53.216223 16416 raft_consensus.cc:729] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb30ddfc6df9492e8b6223d6e9ccd912, State: Initialized, Role: FOLLOWER
I20260924 03:12:53.216815 16416 consensus_queue.cc:260] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b765b6326824826a0d2e92c7859ac00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40015 } } peers { permanent_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42661 } } peers { permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 } }
I20260924 03:12:53.219553 16397 heartbeater.cc:503] Master 127.0.97.62:32789 was elected leader, sending a full tablet report...
I20260924 03:12:53.220109 16417 ts_tablet_manager.cc:1428] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9: Time spent starting tablet: real 0.017s user 0.009s sys 0.008s
I20260924 03:12:53.222368 16416 ts_tablet_manager.cc:1428] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
W20260924 03:12:53.302661 16131 tablet.cc:2369] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:12:53.316191 16265 tablet.cc:2369] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:12:53.325109 16398 tablet.cc:2369] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:12:53.389606 16423 raft_consensus.cc:482] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:12:53.389945 16423 raft_consensus.cc:504] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b765b6326824826a0d2e92c7859ac00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40015 } } peers { permanent_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42661 } } peers { permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 } }
I20260924 03:12:53.391610 16423 leader_election.cc:288] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b765b6326824826a0d2e92c7859ac00 (127.0.97.2:40015), 7371cf39fb024460af08c87a3e1269d9 (127.0.97.3:38127)
I20260924 03:12:53.402101 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:53.392141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54399 (local address 127.0.97.3:38127)
0924 03:12:53.392377 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:53.392384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:53.393982 (+ 1598us) server_negotiation.cc:407] Connection header received
0924 03:12:53.394099 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:53.394105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:53.394188 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:53.394293 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:53.395187 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.396319 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.399396 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.400826 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.400941 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:53.401044 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:53.401740 (+ 696us) server_negotiation.cc:299] Negotiation successful
0924 03:12:53.401861 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 03:12:53.402123 16410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:53.392586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60499 (local address 127.0.97.2:40015)
0924 03:12:53.392771 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:53.392775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:53.393887 (+ 1112us) server_negotiation.cc:407] Connection header received
0924 03:12:53.394100 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:53.394106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:53.394188 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:53.394293 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:53.395187 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.396319 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.399430 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.400771 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.400932 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:53.401041 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:53.401689 (+ 648us) server_negotiation.cc:299] Negotiation successful
0924 03:12:53.401835 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 03:12:53.402315 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:53.392772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40015 (local address 127.0.97.1:60499)
0924 03:12:53.393754 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:53.393776 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:53.393959 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:53.394356 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:53.394362 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:53.394395 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:53.394961 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:53.394971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.396487 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.396510 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:53.399193 (+ 2683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:53.399209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.401197 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.401211 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:53.401439 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:53.401512 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 03:12:53.401626 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 03:12:53.401782 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":224,"thread_start_us":441,"threads_started":1}
I20260924 03:12:53.402784 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:53.392772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38127 (local address 127.0.97.1:54399)
0924 03:12:53.393869 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:53.393885 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:53.393978 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:53.394356 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:53.394362 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:53.394460 (+ 98us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:53.394956 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:53.394971 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.396639 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.396646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:53.399193 (+ 2547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:53.399209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.401226 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.401232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:53.401438 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:53.401597 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 03:12:53.401688 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 03:12:53.401785 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":230,"spinlock_wait_cycles":22400,"thread_start_us":374,"threads_started":1}
I20260924 03:12:53.403548 16219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a0ac2f5ddf4fbeb9d11eb48fa17b8f" candidate_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b765b6326824826a0d2e92c7859ac00" is_pre_election: true
I20260924 03:12:53.404048 16219 raft_consensus.cc:2454] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb30ddfc6df9492e8b6223d6e9ccd912 in term 0.
I20260924 03:12:53.404774 16018 leader_election.cc:302] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b765b6326824826a0d2e92c7859ac00, cb30ddfc6df9492e8b6223d6e9ccd912; no voters:
I20260924 03:12:53.405200 16423 raft_consensus.cc:2781] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:12:53.405375 16423 raft_consensus.cc:482] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:12:53.405269 16352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a0ac2f5ddf4fbeb9d11eb48fa17b8f" candidate_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7371cf39fb024460af08c87a3e1269d9" is_pre_election: true
I20260924 03:12:53.405480 16423 raft_consensus.cc:3037] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:53.405802 16352 raft_consensus.cc:2454] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb30ddfc6df9492e8b6223d6e9ccd912 in term 0.
I20260924 03:12:53.408535 16423 raft_consensus.cc:504] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b765b6326824826a0d2e92c7859ac00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40015 } } peers { permanent_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42661 } } peers { permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 } }
I20260924 03:12:53.409185 16423 leader_election.cc:288] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 [CANDIDATE]: Term 1 election: Requested vote from peers 9b765b6326824826a0d2e92c7859ac00 (127.0.97.2:40015), 7371cf39fb024460af08c87a3e1269d9 (127.0.97.3:38127)
I20260924 03:12:53.409708 16352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a0ac2f5ddf4fbeb9d11eb48fa17b8f" candidate_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7371cf39fb024460af08c87a3e1269d9"
I20260924 03:12:53.409818 16219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a0ac2f5ddf4fbeb9d11eb48fa17b8f" candidate_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b765b6326824826a0d2e92c7859ac00"
I20260924 03:12:53.409924 16352 raft_consensus.cc:3037] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:53.410084 16219 raft_consensus.cc:3037] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:53.413280 16352 raft_consensus.cc:2454] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb30ddfc6df9492e8b6223d6e9ccd912 in term 1.
I20260924 03:12:53.413271 16219 raft_consensus.cc:2454] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb30ddfc6df9492e8b6223d6e9ccd912 in term 1.
I20260924 03:12:53.413865 16019 leader_election.cc:302] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7371cf39fb024460af08c87a3e1269d9, 9b765b6326824826a0d2e92c7859ac00, cb30ddfc6df9492e8b6223d6e9ccd912; no voters:
I20260924 03:12:53.414252 16423 raft_consensus.cc:2781] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:53.414848 16423 raft_consensus.cc:686] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 [term 1 LEADER]: Becoming Leader. State: Replica: cb30ddfc6df9492e8b6223d6e9ccd912, State: Running, Role: LEADER
I20260924 03:12:53.415337 16423 consensus_queue.cc:237] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b765b6326824826a0d2e92c7859ac00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40015 } } peers { permanent_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42661 } } peers { permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 } }
I20260924 03:12:53.419996 15940 catalog_manager.cc:5495] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb30ddfc6df9492e8b6223d6e9ccd912 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b765b6326824826a0d2e92c7859ac00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 } health_report { overall_health: UNKNOWN } } }
I20260924 03:12:53.479167 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:53.473609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37802 (local address 127.0.97.1:42661)
0924 03:12:53.473835 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:53.473840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:53.473858 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:12:53.474059 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:53.474064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:53.474156 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:53.474293 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:12:53.474756 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.475506 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.476978 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.477376 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.477479 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:53.478012 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:12:53.478531 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:12:53.478651 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:53.478779 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 03:12:53.478842 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 03:12:53.488406 16219 raft_consensus.cc:1261] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00 [term 1 FOLLOWER]: Refusing update from remote peer cb30ddfc6df9492e8b6223d6e9ccd912: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:53.488402 16352 raft_consensus.cc:1261] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9 [term 1 FOLLOWER]: Refusing update from remote peer cb30ddfc6df9492e8b6223d6e9ccd912: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:53.489503 16423 consensus_queue.cc:1035] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:53.489789 16426 consensus_queue.cc:1035] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b765b6326824826a0d2e92c7859ac00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:53.518539 15941 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:12:53.518837 15941 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:12:53.522809 15941 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184: Sending DeleteTablet for 3 replicas of tablet c3a0ac2f5ddf4fbeb9d11eb48fa17b8f
I20260924 03:12:53.524057 16065 tablet_service.cc:1514] Processing DeleteTablet for tablet c3a0ac2f5ddf4fbeb9d11eb48fa17b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37798
I20260924 03:12:53.524056 16199 tablet_service.cc:1514] Processing DeleteTablet for tablet c3a0ac2f5ddf4fbeb9d11eb48fa17b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60398
I20260924 03:12:53.524065 16332 tablet_service.cc:1514] Processing DeleteTablet for tablet c3a0ac2f5ddf4fbeb9d11eb48fa17b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32798
I20260924 03:12:53.524374 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 16002
I20260924 03:12:53.537226 16439 tablet_replica.cc:330] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9: stopping tablet replica
I20260924 03:12:53.537914 16439 raft_consensus.cc:2229] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:12:53.538657 16439 raft_consensus.cc:2258] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:53.541181 16441 tablet_replica.cc:330] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00: stopping tablet replica
I20260924 03:12:53.541559 16441 raft_consensus.cc:2229] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:12:53.541909 16439 ts_tablet_manager.cc:1905] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:53.541994 16441 raft_consensus.cc:2258] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:53.546423 16441 ts_tablet_manager.cc:1905] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:53.547034 16440 tablet_replica.cc:330] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912: stopping tablet replica
I20260924 03:12:53.547442 16440 raft_consensus.cc:2229] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:12:53.548089 16440 raft_consensus.cc:2258] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:53.550786 16440 ts_tablet_manager.cc:1905] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:53.550958 16439 ts_tablet_manager.cc:1918] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:53.551143 16439 log.cc:1196] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/c3a0ac2f5ddf4fbeb9d11eb48fa17b8f
I20260924 03:12:53.551667 16439 ts_tablet_manager.cc:1939] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 7371cf39fb024460af08c87a3e1269d9: Deleting consensus metadata
I20260924 03:12:53.553623 15926 catalog_manager.cc:4841] TS 7371cf39fb024460af08c87a3e1269d9 (127.0.97.3:38127): tablet c3a0ac2f5ddf4fbeb9d11eb48fa17b8f (table test-table [id=793dd46a4f604a3aa6800db98253060f]) successfully deleted
I20260924 03:12:53.554960 16441 ts_tablet_manager.cc:1918] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:53.555145 16441 log.cc:1196] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/c3a0ac2f5ddf4fbeb9d11eb48fa17b8f
I20260924 03:12:53.555701 16441 ts_tablet_manager.cc:1939] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P 9b765b6326824826a0d2e92c7859ac00: Deleting consensus metadata
I20260924 03:12:53.557749 15925 catalog_manager.cc:4841] TS 9b765b6326824826a0d2e92c7859ac00 (127.0.97.2:40015): tablet c3a0ac2f5ddf4fbeb9d11eb48fa17b8f (table test-table [id=793dd46a4f604a3aa6800db98253060f]) successfully deleted
Sep 24 03:12:53 dist-test-slave-5l98 krb5kdc[15896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:12:53.560396 16440 ts_tablet_manager.cc:1918] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:53.560629 16440 log.cc:1196] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/c3a0ac2f5ddf4fbeb9d11eb48fa17b8f
I20260924 03:12:53.561205 16440 ts_tablet_manager.cc:1939] T c3a0ac2f5ddf4fbeb9d11eb48fa17b8f P cb30ddfc6df9492e8b6223d6e9ccd912: Deleting consensus metadata
I20260924 03:12:53.563628 15928 catalog_manager.cc:4841] TS cb30ddfc6df9492e8b6223d6e9ccd912 (127.0.97.1:42661): tablet c3a0ac2f5ddf4fbeb9d11eb48fa17b8f (table test-table [id=793dd46a4f604a3aa6800db98253060f]) successfully deleted
I20260924 03:12:53.566942 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:53.525861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37814 (local address 127.0.97.1:42661)
0924 03:12:53.526084 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:53.526093 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:53.537480 (+ 11387us) server_negotiation.cc:407] Connection header received
0924 03:12:53.537871 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:53.537880 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:53.537999 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:53.538094 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:53.542417 (+ 4323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.543237 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.548967 (+ 5730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.549784 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.549922 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:53.559101 (+ 9179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:53.559113 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:53.559135 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:53.559179 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:53.563219 (+ 4040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:53.564521 (+ 1302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:53.564531 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:53.564535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:53.564710 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:53.565148 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:53.565155 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:53.565163 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:53.565620 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:53.565941 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:53.566304 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 03:12:53.566567 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 03:12:53.567755 16125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:53.669925 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 16002
I20260924 03:12:53.689266 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 16136
Sep 24 03:12:53 dist-test-slave-5l98 krb5kdc[15896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 03:12:53.705616 16410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:53.690069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60400 (local address 127.0.97.2:40015)
0924 03:12:53.690266 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:53.690274 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:53.690378 (+ 104us) server_negotiation.cc:407] Connection header received
0924 03:12:53.690665 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:53.690670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:53.690754 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:53.690867 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:53.692149 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.693059 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.694430 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.694808 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.694947 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:53.699812 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:53.699825 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:53.699841 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:53.699874 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:53.702763 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:53.703467 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:53.703475 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:53.703477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:53.703618 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:53.703977 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:53.703980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:53.703982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:53.704354 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:53.704666 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:53.705087 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 03:12:53.705248 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:12:53.706245 16259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:53.798345 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 16136
I20260924 03:12:53.815760 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 16269
Sep 24 03:12:53 dist-test-slave-5l98 krb5kdc[15896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 03:12:53.832230 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:53.816531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32808 (local address 127.0.97.3:38127)
0924 03:12:53.816737 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:53.816744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:53.816894 (+ 150us) server_negotiation.cc:407] Connection header received
0924 03:12:53.817112 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:53.817116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:53.817185 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:53.817292 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:53.818693 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.819422 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.820921 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:53.821312 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:53.821445 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:53.826096 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:53.826112 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:53.826128 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:53.826179 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:53.829335 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:53.830095 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:53.830102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:53.830104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:53.830250 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:53.830690 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:53.830695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:53.830697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:53.831043 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:53.831337 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:53.831652 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 03:12:53.831873 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:12:53.832998 16392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:53.925208 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 16269
I20260924 03:12:53.941774 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 15909
I20260924 03:12:53.942955 15971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:12:54.056131 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 15909
I20260924 03:13:14.077785 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32789
--webserver_interface=127.0.97.62
--webserver_port=33231
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:13:14.426635 16447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:13:14.426934 16447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:13:14.427090 16447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:13:14.434864 16447 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 03:13:14.434978 16447 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 03:13:14.435017 16447 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 03:13:14.435120 16447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:13:14.435197 16447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:13:14.435295 16447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:13:14.435367 16447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:13:14.435451 16447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:13:14.446004 16447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:13:14.447351 16447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:13:14.448845 16447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:13:14.473896 16453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:14.473807 16456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:14.473842 16454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:13:14.478462 16447 server_base.cc:1034] running on GCE node
I20260924 03:13:14.479540 16447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:13:14.481525 16447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:13:14.482805 16447 hybrid_clock.cc:648] HybridClock initialized: now 1790219594482727 us; error 102 us; skew 500 ppm
I20260924 03:13:14.483299 16447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:13:14 dist-test-slave-5l98 krb5kdc[15896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:13:14.494490 16447 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:13:14.496455 16447 webserver.cc:460] Webserver started at http://127.0.97.62:33231/ using document root <none> and password file <none>
I20260924 03:13:14.497321 16447 fs_manager.cc:359] Metadata directory not provided
I20260924 03:13:14.497431 16447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:13:14.503605 16447 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 03:13:14.506668 16463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:14.508287 16447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:13:14.508448 16447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "369d40362da64878af413665ae5a3184"
format_stamp: "Formatted at 2026-09-24 03:12:51 on dist-test-slave-5l98"
server_key: "8226170246af57e1628c4e9bbfa0cdcf"
server_key_iv: "2af8407c6c7bf93e30d0cf3f258b7b28"
server_key_version: "encryptionkey@0"
I20260924 03:13:14.509714 16447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:13:14.529956 16447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:13:14.534976 16447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:13:14.535266 16447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:13:14.557252 16447 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32789
I20260924 03:13:14.557276 16514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32789 every 8 connection(s)
I20260924 03:13:14.558991 16447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:13:14.563089 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 16447
I20260924 03:13:14.564075 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42661
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:32789
--webserver_port=45865
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:13:14.568257 16515 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:13:14.573658 16515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184: Bootstrap starting.
I20260924 03:13:14.594112 16515 log.cc:824] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184: Log is configured to *not* fsync() on all Append() calls
I20260924 03:13:14.606280 16515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:13:14.606879 16515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184: Bootstrap complete.
I20260924 03:13:14.616840 16515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369d40362da64878af413665ae5a3184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32789 } }
I20260924 03:13:14.617703 16515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 369d40362da64878af413665ae5a3184, State: Initialized, Role: FOLLOWER
I20260924 03:13:14.618346 16515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369d40362da64878af413665ae5a3184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32789 } }
I20260924 03:13:14.618587 16515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:13:14.618716 16515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:13:14.618902 16515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:13:14.622917 16515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369d40362da64878af413665ae5a3184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32789 } }
I20260924 03:13:14.623441 16515 leader_election.cc:302] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 369d40362da64878af413665ae5a3184; no voters:
I20260924 03:13:14.624185 16515 leader_election.cc:288] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 03:13:14.624351 16519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:13:14.625633 16519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 2 LEADER]: Becoming Leader. State: Replica: 369d40362da64878af413665ae5a3184, State: Running, Role: LEADER
I20260924 03:13:14.626296 16519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369d40362da64878af413665ae5a3184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32789 } }
I20260924 03:13:14.626546 16515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:13:14.629067 16521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 369d40362da64878af413665ae5a3184. Latest consensus state: current_term: 2 leader_uuid: "369d40362da64878af413665ae5a3184" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369d40362da64878af413665ae5a3184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32789 } } }
I20260924 03:13:14.629457 16521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [sys.catalog]: This master's current role is: LEADER
I20260924 03:13:14.629181 16520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "369d40362da64878af413665ae5a3184" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369d40362da64878af413665ae5a3184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32789 } } }
I20260924 03:13:14.629633 16520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [sys.catalog]: This master's current role is: LEADER
I20260924 03:13:14.638024 16526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:13:14.648476 16526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:13:14.650528 16526 catalog_manager.cc:1259] Loaded cluster ID: 08a9c020672f496cb5d322c663646fee
I20260924 03:13:14.650772 16526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:13:14.656307 16526 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:13:14.658526 16526 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184: Loaded TSK: 0
I20260924 03:13:14.714437 16526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184: Generated new TSK 1
I20260924 03:13:14.714974 16526 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:13:14.980381 16517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:13:14.980786 16517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:13:14.980965 16517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:13:14.989070 16517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:13:14.989210 16517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:13:14.989410 16517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:13:15.001142 16517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45865
--tserver_master_addrs=127.0.97.62:32789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:13:15.002454 16517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:13:15.004091 16517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:13:15.021934 16542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:15.021914 16543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:15.022120 16545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:13:15.022455 16517 server_base.cc:1034] running on GCE node
I20260924 03:13:15.023413 16517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:13:15.024917 16517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:13:15.026291 16517 hybrid_clock.cc:648] HybridClock initialized: now 1790219595026234 us; error 68 us; skew 500 ppm
I20260924 03:13:15.026719 16517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:13:15 dist-test-slave-5l98 krb5kdc[15896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:13:15.031060 16517 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:13:15.033026 16517 webserver.cc:460] Webserver started at http://127.0.97.1:45865/ using document root <none> and password file <none>
I20260924 03:13:15.033746 16517 fs_manager.cc:359] Metadata directory not provided
I20260924 03:13:15.033895 16517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:13:15.040081 16517 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 03:13:15.043035 16552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:15.044646 16517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 03:13:15.044831 16517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "cb30ddfc6df9492e8b6223d6e9ccd912"
format_stamp: "Formatted at 2026-09-24 03:12:51 on dist-test-slave-5l98"
server_key: "e50e433e8125fcb44e63e74c921a8eeb"
server_key_iv: "39a56c65beed25b89c173e247772a434"
server_key_version: "encryptionkey@0"
I20260924 03:13:15.045853 16517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:13:15.070554 16517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:13:15.075809 16517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:13:15.076134 16517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:13:15.077227 16517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:13:15.078722 16517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:13:15.078843 16517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:15.078979 16517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:13:15.079066 16517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:15.114727 16517 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42661
I20260924 03:13:15.114830 16664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42661 every 8 connection(s)
I20260924 03:13:15.116230 16517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:13:15.123723 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 16517
I20260924 03:13:15.124640 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40015
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:32789
--webserver_port=43479
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:13:15 dist-test-slave-5l98 krb5kdc[15896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:13:15.146068 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:15.119332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53507 (local address 127.0.97.62:32789)
0924 03:13:15.120117 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:15.120129 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:15.120914 (+ 785us) server_negotiation.cc:407] Connection header received
0924 03:13:15.123935 (+ 3021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:15.123960 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:15.124456 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:15.125029 (+ 573us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:13:15.127692 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:15.129162 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:15.131703 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:15.133102 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:15.133330 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:15.137467 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:13:15.137480 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:13:15.137510 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:13:15.137562 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:13:15.140954 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:15.142065 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:15.142072 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:15.142077 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:15.142265 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:15.142864 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:15.142868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:15.142870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:15.143409 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:13:15.143560 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:15.144736 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0924 03:13:15.145074 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":468,"threads_started":1}
I20260924 03:13:15.146515 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:15.119795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.1:53507)
0924 03:13:15.120734 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:15.120779 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:13:15.123252 (+ 2473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:13:15.125760 (+ 2508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:13:15.125774 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:13:15.126587 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:13:15.127414 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:15.127443 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:15.129350 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:15.129359 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:15.131509 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:15.131528 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:15.133342 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:15.133347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:15.133540 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:15.134366 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:13:15.134389 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:13:15.137157 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:13:15.141135 (+ 3978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:13:15.141148 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:13:15.141168 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:13:15.141866 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:13:15.142417 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:13:15.142425 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:13:15.142432 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:13:15.142695 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:13:15.143583 (+ 888us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:13:15.143594 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:13:15.144064 (+ 470us) client_negotiation.cc:769] Sending connection context
0924 03:13:15.144418 (+ 354us) client_negotiation.cc:240] Negotiation successful
0924 03:13:15.145093 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":399,"threads_started":1}
I20260924 03:13:15.149927 16665 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32789
I20260924 03:13:15.150297 16665 heartbeater.cc:464] Registering TS with master...
I20260924 03:13:15.151149 16665 heartbeater.cc:511] Master 127.0.97.62:32789 requested a full tablet report, sending...
I20260924 03:13:15.153424 16480 ts_manager.cc:194] Registered new tserver with Master: cb30ddfc6df9492e8b6223d6e9ccd912 (127.0.97.1:42661)
I20260924 03:13:15.154932 16480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53507
W20260924 03:13:15.463693 16669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:13:15.464002 16669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:13:15.464156 16669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:13:15.472270 16669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:13:15.472424 16669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:13:15.472661 16669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:13:15.482849 16669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43479
--tserver_master_addrs=127.0.97.62:32789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:13:15.484163 16669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:13:15.485507 16669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:13:15.499261 16675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:15.499311 16674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:15.499892 16677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:13:15.500142 16669 server_base.cc:1034] running on GCE node
I20260924 03:13:15.500813 16669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:13:15.501796 16669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:13:15.503029 16669 hybrid_clock.cc:648] HybridClock initialized: now 1790219595502967 us; error 54 us; skew 500 ppm
I20260924 03:13:15.503386 16669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:13:15 dist-test-slave-5l98 krb5kdc[15896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:13:15.506625 16669 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:13:15.508183 16669 webserver.cc:460] Webserver started at http://127.0.97.2:43479/ using document root <none> and password file <none>
I20260924 03:13:15.508940 16669 fs_manager.cc:359] Metadata directory not provided
I20260924 03:13:15.509116 16669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:13:15.514881 16669 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 03:13:15.517993 16684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:15.519374 16669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:13:15.519666 16669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "9b765b6326824826a0d2e92c7859ac00"
format_stamp: "Formatted at 2026-09-24 03:12:52 on dist-test-slave-5l98"
server_key: "9a54bed913ba8bc2dbf23a8945558336"
server_key_iv: "864c70f9bdc29919e6f2a3f047da0643"
server_key_version: "encryptionkey@0"
I20260924 03:13:15.520691 16669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:13:15.534520 16669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:13:15.539155 16669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:13:15.539474 16669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:13:15.540640 16669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:13:15.541998 16669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:13:15.542125 16669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:15.542270 16669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:13:15.542348 16669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:15.578747 16669 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40015
I20260924 03:13:15.578799 16796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40015 every 8 connection(s)
I20260924 03:13:15.580108 16669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:13:15.582736 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 16669
I20260924 03:13:15.583426 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:38127
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:32789
--webserver_port=45603
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:13:15 dist-test-slave-5l98 krb5kdc[15896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:13:15.606736 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:15.583008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33681 (local address 127.0.97.62:32789)
0924 03:13:15.583217 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:15.583224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:15.584569 (+ 1345us) server_negotiation.cc:407] Connection header received
0924 03:13:15.587489 (+ 2920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:15.587494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:15.587594 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:15.587687 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:13:15.589956 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:15.590886 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:15.592725 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:15.593676 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:15.593845 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:15.598079 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:13:15.598089 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:13:15.598096 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:13:15.598151 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:13:15.602345 (+ 4194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:15.603209 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:15.603219 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:15.603226 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:15.603409 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:15.603971 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:15.603980 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:15.603994 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:15.604460 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:13:15.604673 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:15.605855 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0924 03:13:15.606205 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 03:13:15.607679 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:15.583411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.2:33681)
0924 03:13:15.584318 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:15.584372 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 03:13:15.587126 (+ 2754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:13:15.587978 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:13:15.587992 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:13:15.588891 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:13:15.589726 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:15.589754 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:15.591060 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:15.591068 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:15.592563 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:15.592576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:15.593936 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:15.593940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:15.594128 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:15.594868 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:13:15.594890 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:13:15.597742 (+ 2852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:13:15.602527 (+ 4785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:13:15.602535 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:13:15.602551 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:13:15.603040 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:13:15.603551 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:13:15.603555 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:13:15.603557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:13:15.603809 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:13:15.604765 (+ 956us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:13:15.604773 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:13:15.605391 (+ 618us) client_negotiation.cc:769] Sending connection context
0924 03:13:15.605677 (+ 286us) client_negotiation.cc:240] Negotiation successful
0924 03:13:15.606173 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":331,"threads_started":1}
I20260924 03:13:15.609673 16797 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32789
I20260924 03:13:15.610097 16797 heartbeater.cc:464] Registering TS with master...
I20260924 03:13:15.611207 16797 heartbeater.cc:511] Master 127.0.97.62:32789 requested a full tablet report, sending...
I20260924 03:13:15.613154 16480 ts_manager.cc:194] Registered new tserver with Master: 9b765b6326824826a0d2e92c7859ac00 (127.0.97.2:40015)
I20260924 03:13:15.614368 16480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33681
W20260924 03:13:15.925938 16800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:13:15.926223 16800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:13:15.926386 16800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:13:15.933669 16800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:13:15.933781 16800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:13:15.933960 16800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:13:15.943802 16800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:38127
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45603
--tserver_master_addrs=127.0.97.62:32789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:13:15.945164 16800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:13:15.946398 16800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:13:15.960369 16805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:15.960415 16806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:15.960469 16808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:13:15.961334 16800 server_base.cc:1034] running on GCE node
I20260924 03:13:15.961947 16800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:13:15.962926 16800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:13:15.964160 16800 hybrid_clock.cc:648] HybridClock initialized: now 1790219595964069 us; error 69 us; skew 500 ppm
I20260924 03:13:15.964615 16800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:13:15 dist-test-slave-5l98 krb5kdc[15896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:13:15.967624 16800 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:13:15.969226 16800 webserver.cc:460] Webserver started at http://127.0.97.3:45603/ using document root <none> and password file <none>
I20260924 03:13:15.969901 16800 fs_manager.cc:359] Metadata directory not provided
I20260924 03:13:15.970012 16800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:13:15.975343 16800 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 03:13:15.978248 16815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:15.979583 16800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:13:15.979775 16800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "7371cf39fb024460af08c87a3e1269d9"
format_stamp: "Formatted at 2026-09-24 03:12:52 on dist-test-slave-5l98"
server_key: "d8337a2a1868243069a8139c031e4ad0"
server_key_iv: "fa40babc994cf5fda7c4d13f1ccc3593"
server_key_version: "encryptionkey@0"
I20260924 03:13:15.980871 16800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:13:15.994453 16800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:13:15.999276 16800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:13:15.999624 16800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:13:16.001039 16800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:13:16.002876 16800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:13:16.002965 16800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:16.003041 16800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:13:16.003088 16800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:16.039054 16800 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38127
I20260924 03:13:16.039150 16927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38127 every 8 connection(s)
I20260924 03:13:16.040572 16800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:13:16.044334 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 16800
Sep 24 03:13:16 dist-test-slave-5l98 krb5kdc[15896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:13:16.062678 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.045268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59858 (local address 127.0.97.62:32789)
0924 03:13:16.045750 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:16.045756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:16.046230 (+ 474us) server_negotiation.cc:407] Connection header received
0924 03:13:16.046406 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:16.046410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:16.046479 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:16.046562 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:13:16.048704 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.049843 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.051917 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.052430 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.052574 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:16.055982 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:13:16.055994 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:13:16.055997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:13:16.056028 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:13:16.059869 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:16.060566 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:16.060573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:16.060579 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:16.060715 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:16.061107 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:16.061110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:16.061112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:16.061428 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:13:16.061666 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:16.062094 (+ 428us) server_negotiation.cc:299] Negotiation successful
0924 03:13:16.062314 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":271,"threads_started":1}
I20260924 03:13:16.063995 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.043075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52335 (local address 127.0.97.62:32789)
0924 03:13:16.043303 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:16.043310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:16.044688 (+ 1378us) server_negotiation.cc:407] Connection header received
0924 03:13:16.047194 (+ 2506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:16.047197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:16.047240 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:16.047301 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:13:16.049268 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.050212 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.052313 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.052932 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.053018 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:16.057301 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:13:16.057309 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:13:16.057318 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:13:16.057366 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:13:16.060464 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:16.061372 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:16.061376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:16.061380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:16.061491 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:16.061985 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:16.061990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:16.061992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:16.062314 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:13:16.062453 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:16.063438 (+ 985us) server_negotiation.cc:299] Negotiation successful
0924 03:13:16.063666 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 03:13:16.064945 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.043561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.3:52335)
0924 03:13:16.044472 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:16.044548 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0924 03:13:16.046906 (+ 2358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:13:16.047573 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:13:16.047588 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:13:16.048322 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:13:16.049110 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:16.049129 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.050351 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.050357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:16.052183 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:16.052199 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.053196 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.053202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:16.053466 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:16.054160 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:13:16.054194 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:13:16.057049 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:13:16.060654 (+ 3605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:13:16.060661 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:13:16.060673 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:13:16.061253 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:13:16.061624 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:13:16.061629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:13:16.061634 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:13:16.061870 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:13:16.062527 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:13:16.062542 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:13:16.062938 (+ 396us) client_negotiation.cc:769] Sending connection context
0924 03:13:16.063198 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 03:13:16.063666 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":363,"threads_started":1}
I20260924 03:13:16.066402 16928 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32789
I20260924 03:13:16.066705 16928 heartbeater.cc:464] Registering TS with master...
I20260924 03:13:16.067405 16928 heartbeater.cc:511] Master 127.0.97.62:32789 requested a full tablet report, sending...
I20260924 03:13:16.068974 16480 ts_manager.cc:194] Registered new tserver with Master: 7371cf39fb024460af08c87a3e1269d9 (127.0.97.3:38127)
I20260924 03:13:16.070024 16480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52335
I20260924 03:13:16.077724 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:13:16.084688 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.078648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59860 (local address 127.0.97.62:32789)
0924 03:13:16.078912 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:16.078919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:16.079252 (+ 333us) server_negotiation.cc:407] Connection header received
0924 03:13:16.079496 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:16.079500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:16.079628 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:16.079785 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:13:16.080244 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.081148 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.082620 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.083054 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.083197 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:16.083726 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:13:16.083925 (+ 199us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 03:13:16.084340 (+ 415us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59860: token expired
Metrics: {"server-negotiator.queue_time_us":110}
W20260924 03:13:16.084823 16667 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59860: token expired
W20260924 03:13:16.085774 16402 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:32789)
I20260924 03:13:16.099872 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.086685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59862 (local address 127.0.97.62:32789)
0924 03:13:16.086882 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:16.086888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:16.086905 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:13:16.087013 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:16.087017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:16.087092 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:16.087194 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:13:16.088604 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.089409 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.090797 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.091173 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.091305 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:16.094045 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:13:16.094049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:13:16.094053 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:13:16.094083 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:13:16.096958 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:16.097707 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:16.097714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:16.097723 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:16.097840 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:16.098236 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:16.098243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:16.098246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:16.098552 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:13:16.098820 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:16.099270 (+ 450us) server_negotiation.cc:299] Negotiation successful
0924 03:13:16.099518 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 03:13:16.103437 16480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:13:16.110852 16480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:13:16.145741 16940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.123853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54400 (local address 127.0.97.3:38127)
0924 03:13:16.138015 (+ 14162us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:16.138024 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:16.138058 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:13:16.138174 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:16.138183 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:16.138403 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:16.139195 (+ 792us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:13:16.139473 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.141139 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.143550 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.144539 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.144706 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:16.144947 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:16.145208 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 03:13:16.145371 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13995,"thread_start_us":13830,"threads_started":1}
I20260924 03:13:16.145859 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.123439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38127 (local address 127.0.0.1:54400)
0924 03:13:16.129671 (+ 6232us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:16.129744 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 03:13:16.129951 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:13:16.138612 (+ 8661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:13:16.138616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:13:16.138639 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:13:16.138985 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:16.138996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.141303 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.141309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:16.143390 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:16.143402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.144757 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.144764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:16.144957 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:16.145019 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 03:13:16.145160 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 03:13:16.145362 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5884,"mutex_wait_us":46,"thread_start_us":5702,"threads_started":1}
I20260924 03:13:16.146548 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.123932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40015 (local address 127.0.0.1:54866)
0924 03:13:16.134631 (+ 10699us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:16.134650 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:13:16.134763 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:13:16.137356 (+ 2593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:13:16.137368 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:13:16.137415 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:13:16.138269 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:16.138283 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.140031 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.140037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:16.142440 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:16.142455 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.145655 (+ 3200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.145663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:16.145870 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:16.145900 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 03:13:16.146004 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 03:13:16.146140 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10568,"mutex_wait_us":456,"thread_start_us":10512,"threads_started":1}
I20260924 03:13:16.147377 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.124136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54866 (local address 127.0.97.2:40015)
0924 03:13:16.136207 (+ 12071us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:16.136219 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:16.136255 (+ 36us) server_negotiation.cc:407] Connection header received
0924 03:13:16.136374 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:16.136386 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:16.136765 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:16.137000 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:13:16.138506 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.139912 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.143582 (+ 3670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.145080 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.145208 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:16.145381 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:16.146860 (+ 1479us) server_negotiation.cc:299] Negotiation successful
0924 03:13:16.147046 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11865,"thread_start_us":11732,"threads_started":1}
I20260924 03:13:16.147388 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.124285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42661 (local address 127.0.0.1:50018)
0924 03:13:16.138353 (+ 14068us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:16.138372 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:13:16.138499 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:13:16.139212 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:13:16.139217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:13:16.139240 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:13:16.139548 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:16.139558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.141415 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.141420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:16.143407 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:16.143419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.145667 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.145671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:16.146418 (+ 747us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:16.146449 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 03:13:16.146553 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 03:13:16.146993 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13908,"mutex_wait_us":34,"thread_start_us":13184,"threads_started":1}
I20260924 03:13:16.147548 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.125008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50018 (local address 127.0.97.1:42661)
0924 03:13:16.137105 (+ 12097us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:16.137118 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:16.138632 (+ 1514us) server_negotiation.cc:407] Connection header received
0924 03:13:16.138732 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:16.138739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:16.139012 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:16.139191 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:13:16.139709 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.141235 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.143586 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.145080 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.145221 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:16.145414 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:16.147026 (+ 1612us) server_negotiation.cc:299] Negotiation successful
0924 03:13:16.147206 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11846,"thread_start_us":11693,"threads_started":1}
I20260924 03:13:16.150233 16863 tablet_service.cc:1467] Processing CreateTablet for tablet 0898eb33a4524a4c919663281fe4532e (DEFAULT_TABLE table=test-table [id=b8cf3c5831f34d5a925e4ea9a42be571]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:13:16.150998 16600 tablet_service.cc:1467] Processing CreateTablet for tablet 0898eb33a4524a4c919663281fe4532e (DEFAULT_TABLE table=test-table [id=b8cf3c5831f34d5a925e4ea9a42be571]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:13:16.151186 16732 tablet_service.cc:1467] Processing CreateTablet for tablet 0898eb33a4524a4c919663281fe4532e (DEFAULT_TABLE table=test-table [id=b8cf3c5831f34d5a925e4ea9a42be571]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:13:16.151738 16863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0898eb33a4524a4c919663281fe4532e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:13:16.152436 16732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0898eb33a4524a4c919663281fe4532e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:13:16.152657 16600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0898eb33a4524a4c919663281fe4532e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:13:16.158349 16665 heartbeater.cc:503] Master 127.0.97.62:32789 was elected leader, sending a full tablet report...
I20260924 03:13:16.165212 16944 tablet_bootstrap.cc:492] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912: Bootstrap starting.
I20260924 03:13:16.167405 16945 tablet_bootstrap.cc:492] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9: Bootstrap starting.
I20260924 03:13:16.169082 16946 tablet_bootstrap.cc:492] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00: Bootstrap starting.
I20260924 03:13:16.170401 16944 tablet_bootstrap.cc:654] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912: Neither blocks nor log segments found. Creating new log.
I20260924 03:13:16.171818 16944 log.cc:824] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912: Log is configured to *not* fsync() on all Append() calls
I20260924 03:13:16.172730 16946 tablet_bootstrap.cc:654] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00: Neither blocks nor log segments found. Creating new log.
I20260924 03:13:16.172828 16945 tablet_bootstrap.cc:654] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9: Neither blocks nor log segments found. Creating new log.
I20260924 03:13:16.173961 16945 log.cc:824] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9: Log is configured to *not* fsync() on all Append() calls
I20260924 03:13:16.174005 16946 log.cc:824] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00: Log is configured to *not* fsync() on all Append() calls
I20260924 03:13:16.175951 16944 tablet_bootstrap.cc:492] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912: No bootstrap required, opened a new log
I20260924 03:13:16.176301 16944 ts_tablet_manager.cc:1397] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260924 03:13:16.179158 16946 tablet_bootstrap.cc:492] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00: No bootstrap required, opened a new log
I20260924 03:13:16.179275 16945 tablet_bootstrap.cc:492] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9: No bootstrap required, opened a new log
I20260924 03:13:16.179432 16946 ts_tablet_manager.cc:1397] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20260924 03:13:16.179563 16945 ts_tablet_manager.cc:1397] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20260924 03:13:16.184217 16944 raft_consensus.cc:348] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 } } peers { permanent_uuid: "9b765b6326824826a0d2e92c7859ac00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40015 } } peers { permanent_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42661 } }
I20260924 03:13:16.184698 16944 raft_consensus.cc:374] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:13:16.184835 16944 raft_consensus.cc:729] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb30ddfc6df9492e8b6223d6e9ccd912, State: Initialized, Role: FOLLOWER
I20260924 03:13:16.185536 16944 consensus_queue.cc:260] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 } } peers { permanent_uuid: "9b765b6326824826a0d2e92c7859ac00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40015 } } peers { permanent_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42661 } }
I20260924 03:13:16.189309 16946 raft_consensus.cc:348] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 } } peers { permanent_uuid: "9b765b6326824826a0d2e92c7859ac00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40015 } } peers { permanent_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42661 } }
I20260924 03:13:16.189739 16946 raft_consensus.cc:374] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:13:16.189860 16946 raft_consensus.cc:729] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b765b6326824826a0d2e92c7859ac00, State: Initialized, Role: FOLLOWER
I20260924 03:13:16.190506 16946 consensus_queue.cc:260] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 } } peers { permanent_uuid: "9b765b6326824826a0d2e92c7859ac00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40015 } } peers { permanent_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42661 } }
I20260924 03:13:16.191730 16797 heartbeater.cc:503] Master 127.0.97.62:32789 was elected leader, sending a full tablet report...
I20260924 03:13:16.191596 16945 raft_consensus.cc:348] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 } } peers { permanent_uuid: "9b765b6326824826a0d2e92c7859ac00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40015 } } peers { permanent_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42661 } }
I20260924 03:13:16.192083 16945 raft_consensus.cc:374] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:13:16.192219 16945 raft_consensus.cc:729] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7371cf39fb024460af08c87a3e1269d9, State: Initialized, Role: FOLLOWER
I20260924 03:13:16.192896 16945 consensus_queue.cc:260] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 } } peers { permanent_uuid: "9b765b6326824826a0d2e92c7859ac00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40015 } } peers { permanent_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42661 } }
I20260924 03:13:16.194573 16944 ts_tablet_manager.cc:1428] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912: Time spent starting tablet: real 0.018s user 0.010s sys 0.003s
I20260924 03:13:16.197795 16946 ts_tablet_manager.cc:1428] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00: Time spent starting tablet: real 0.018s user 0.009s sys 0.005s
I20260924 03:13:16.198326 16928 heartbeater.cc:503] Master 127.0.97.62:32789 was elected leader, sending a full tablet report...
I20260924 03:13:16.198956 16945 ts_tablet_manager.cc:1428] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9: Time spent starting tablet: real 0.019s user 0.014s sys 0.003s
I20260924 03:13:16.242101 16951 raft_consensus.cc:482] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:13:16.242460 16951 raft_consensus.cc:504] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 } } peers { permanent_uuid: "9b765b6326824826a0d2e92c7859ac00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40015 } } peers { permanent_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42661 } }
I20260924 03:13:16.244344 16951 leader_election.cc:288] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7371cf39fb024460af08c87a3e1269d9 (127.0.97.3:38127), cb30ddfc6df9492e8b6223d6e9ccd912 (127.0.97.1:42661)
I20260924 03:13:16.251579 16940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.244857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60781 (local address 127.0.97.3:38127)
0924 03:13:16.245235 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:16.245239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:16.245948 (+ 709us) server_negotiation.cc:407] Connection header received
0924 03:13:16.246200 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:16.246205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:16.246267 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:16.246367 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:13:16.246937 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.247632 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.249661 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.250645 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.250788 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:16.250905 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:16.251182 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 03:13:16.251309 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260924 03:13:16.251693 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.245037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38127 (local address 127.0.97.2:60781)
0924 03:13:16.245824 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:16.245840 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:13:16.246032 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:13:16.246383 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:13:16.246388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:13:16.246414 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:13:16.246795 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:16.246804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.247768 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.247774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:16.249503 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:16.249518 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.250853 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.250859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:16.251019 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:16.251044 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 03:13:16.251155 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 03:13:16.251289 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":450,"threads_started":1}
I20260924 03:13:16.252353 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.244857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54273 (local address 127.0.97.1:42661)
0924 03:13:16.245102 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:16.245106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:16.246471 (+ 1365us) server_negotiation.cc:407] Connection header received
0924 03:13:16.246733 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:16.246738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:16.246802 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:16.246909 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:13:16.247602 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.248312 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.250515 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.251272 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.251430 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:16.251542 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:16.252026 (+ 484us) server_negotiation.cc:299] Negotiation successful
0924 03:13:16.252171 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 03:13:16.252389 16954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.245073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42661 (local address 127.0.97.2:54273)
0924 03:13:16.246343 (+ 1270us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:16.246359 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:13:16.246565 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:13:16.246920 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:13:16.246925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:13:16.246948 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:13:16.247409 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:16.247422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.248447 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.248455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:16.250307 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:16.250327 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.251510 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.251516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:16.251737 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:16.251787 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 03:13:16.251881 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 03:13:16.251997 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"spinlock_wait_cycles":47744,"thread_start_us":928,"threads_started":1}
I20260924 03:13:16.252779 16883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0898eb33a4524a4c919663281fe4532e" candidate_uuid: "9b765b6326824826a0d2e92c7859ac00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7371cf39fb024460af08c87a3e1269d9" is_pre_election: true
I20260924 03:13:16.253327 16883 raft_consensus.cc:2454] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b765b6326824826a0d2e92c7859ac00 in term 0.
I20260924 03:13:16.253427 16620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0898eb33a4524a4c919663281fe4532e" candidate_uuid: "9b765b6326824826a0d2e92c7859ac00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" is_pre_election: true
I20260924 03:13:16.253925 16620 raft_consensus.cc:2454] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b765b6326824826a0d2e92c7859ac00 in term 0.
I20260924 03:13:16.253983 16686 leader_election.cc:302] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7371cf39fb024460af08c87a3e1269d9, 9b765b6326824826a0d2e92c7859ac00; no voters:
I20260924 03:13:16.254362 16951 raft_consensus.cc:2781] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:13:16.254520 16951 raft_consensus.cc:482] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:13:16.254647 16951 raft_consensus.cc:3037] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:13:16.256938 16951 raft_consensus.cc:504] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 } } peers { permanent_uuid: "9b765b6326824826a0d2e92c7859ac00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40015 } } peers { permanent_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42661 } }
I20260924 03:13:16.257589 16951 leader_election.cc:288] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 [CANDIDATE]: Term 1 election: Requested vote from peers 7371cf39fb024460af08c87a3e1269d9 (127.0.97.3:38127), cb30ddfc6df9492e8b6223d6e9ccd912 (127.0.97.1:42661)
I20260924 03:13:16.258097 16883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0898eb33a4524a4c919663281fe4532e" candidate_uuid: "9b765b6326824826a0d2e92c7859ac00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7371cf39fb024460af08c87a3e1269d9"
I20260924 03:13:16.258270 16620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0898eb33a4524a4c919663281fe4532e" candidate_uuid: "9b765b6326824826a0d2e92c7859ac00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912"
I20260924 03:13:16.258359 16883 raft_consensus.cc:3037] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:13:16.258524 16620 raft_consensus.cc:3037] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:13:16.260792 16883 raft_consensus.cc:2454] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b765b6326824826a0d2e92c7859ac00 in term 1.
I20260924 03:13:16.260967 16620 raft_consensus.cc:2454] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b765b6326824826a0d2e92c7859ac00 in term 1.
I20260924 03:13:16.261191 16686 leader_election.cc:302] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7371cf39fb024460af08c87a3e1269d9, 9b765b6326824826a0d2e92c7859ac00; no voters:
I20260924 03:13:16.261507 16951 raft_consensus.cc:2781] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:13:16.262053 16951 raft_consensus.cc:686] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 [term 1 LEADER]: Becoming Leader. State: Replica: 9b765b6326824826a0d2e92c7859ac00, State: Running, Role: LEADER
I20260924 03:13:16.262558 16951 consensus_queue.cc:237] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 } } peers { permanent_uuid: "9b765b6326824826a0d2e92c7859ac00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40015 } } peers { permanent_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42661 } }
I20260924 03:13:16.267623 16480 catalog_manager.cc:5495] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b765b6326824826a0d2e92c7859ac00 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "9b765b6326824826a0d2e92c7859ac00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b765b6326824826a0d2e92c7859ac00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42661 } health_report { overall_health: UNKNOWN } } }
W20260924 03:13:16.292353 16929 tablet.cc:2369] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:13:16.305045 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.299477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50978 (local address 127.0.97.2:40015)
0924 03:13:16.299736 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:16.299741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:16.299759 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:13:16.299854 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:16.299858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:16.299940 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:16.300075 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:13:16.300460 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.301357 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.302783 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.303210 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.303309 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:16.303876 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:13:16.304411 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:13:16.304719 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:16.304764 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 03:13:16.304829 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:13:16.313521 16620 raft_consensus.cc:1261] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912 [term 1 FOLLOWER]: Refusing update from remote peer 9b765b6326824826a0d2e92c7859ac00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:13:16.313570 16883 raft_consensus.cc:1261] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9 [term 1 FOLLOWER]: Refusing update from remote peer 9b765b6326824826a0d2e92c7859ac00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:13:16.314591 16955 consensus_queue.cc:1035] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7371cf39fb024460af08c87a3e1269d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:13:16.315110 16951 consensus_queue.cc:1035] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb30ddfc6df9492e8b6223d6e9ccd912" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:13:16.324065 16958 mvcc.cc:205] Tried to move back new op lower bound from 7332739466490429440 to 7332739466295263232. Current Snapshot: MvccSnapshot[applied={T|T < 7332739466490429440}]
I20260924 03:13:16.327121 16959 mvcc.cc:205] Tried to move back new op lower bound from 7332739466490429440 to 7332739466295263232. Current Snapshot: MvccSnapshot[applied={T|T < 7332739466490429440}]
I20260924 03:13:16.346486 16479 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:13:16.346841 16479 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:13:16.350915 16479 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184: Sending DeleteTablet for 3 replicas of tablet 0898eb33a4524a4c919663281fe4532e
I20260924 03:13:16.352347 16863 tablet_service.cc:1514] Processing DeleteTablet for tablet 0898eb33a4524a4c919663281fe4532e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:13:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54400
I20260924 03:13:16.352357 16600 tablet_service.cc:1514] Processing DeleteTablet for tablet 0898eb33a4524a4c919663281fe4532e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:13:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50018
I20260924 03:13:16.352346 16732 tablet_service.cc:1514] Processing DeleteTablet for tablet 0898eb33a4524a4c919663281fe4532e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:13:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54866
I20260924 03:13:16.352605 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 16517
I20260924 03:13:16.367528 16970 tablet_replica.cc:330] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912: stopping tablet replica
I20260924 03:13:16.368258 16968 tablet_replica.cc:330] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00: stopping tablet replica
I20260924 03:13:16.368947 16968 raft_consensus.cc:2229] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:13:16.369534 16969 tablet_replica.cc:330] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9: stopping tablet replica
I20260924 03:13:16.369951 16969 raft_consensus.cc:2229] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:13:16.369957 16970 raft_consensus.cc:2229] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:13:16.370527 16969 raft_consensus.cc:2258] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:13:16.370539 16970 raft_consensus.cc:2258] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:13:16.370946 16968 raft_consensus.cc:2258] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:13:16.374191 16968 ts_tablet_manager.cc:1905] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:13:16.374408 16969 ts_tablet_manager.cc:1905] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:13:16.379573 16970 ts_tablet_manager.cc:1905] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:13:16.381625 16969 ts_tablet_manager.cc:1918] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:13:16.381799 16969 log.cc:1196] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/0898eb33a4524a4c919663281fe4532e
I20260924 03:13:16.382359 16969 ts_tablet_manager.cc:1939] T 0898eb33a4524a4c919663281fe4532e P 7371cf39fb024460af08c87a3e1269d9: Deleting consensus metadata
I20260924 03:13:16.383319 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.354306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51428 (local address 127.0.97.1:42661)
0924 03:13:16.354486 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:16.354492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:16.354513 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:13:16.354574 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:16.354580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:16.354663 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:16.354785 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:13:16.357155 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.358327 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.360298 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.361003 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.361132 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:16.367316 (+ 6184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:13:16.367338 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:13:16.367359 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:13:16.367411 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:13:16.372191 (+ 4780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:16.373238 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:16.373246 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:16.373250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:16.373420 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:16.380530 (+ 7110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:16.380535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:16.380539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:16.381051 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:13:16.381533 (+ 482us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:16.382669 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0924 03:13:16.382914 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:13:16.384536 16968 ts_tablet_manager.cc:1918] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:13:16.384784 16968 log.cc:1196] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/0898eb33a4524a4c919663281fe4532e
I20260924 03:13:16.384941 16465 catalog_manager.cc:4841] TS 7371cf39fb024460af08c87a3e1269d9 (127.0.97.3:38127): tablet 0898eb33a4524a4c919663281fe4532e (table test-table [id=b8cf3c5831f34d5a925e4ea9a42be571]) successfully deleted
I20260924 03:13:16.385485 16968 ts_tablet_manager.cc:1939] T 0898eb33a4524a4c919663281fe4532e P 9b765b6326824826a0d2e92c7859ac00: Deleting consensus metadata
I20260924 03:13:16.385583 16660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:13:16.387727 16464 catalog_manager.cc:4841] TS 9b765b6326824826a0d2e92c7859ac00 (127.0.97.2:40015): tablet 0898eb33a4524a4c919663281fe4532e (table test-table [id=b8cf3c5831f34d5a925e4ea9a42be571]) successfully deleted
I20260924 03:13:16.477820 16970 ts_tablet_manager.cc:1918] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:13:16.478082 16970 log.cc:1196] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/0898eb33a4524a4c919663281fe4532e
I20260924 03:13:16.478466 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 16517
W20260924 03:13:16.497167 16467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42661 (error 108)
I20260924 03:13:16.497287 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 16669
I20260924 03:13:16.498998 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.498571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42661 (local address 127.0.0.1:51442)
0924 03:13:16.498751 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:16.498864 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 03:13:16.499326 16467 catalog_manager.cc:4584] TS cb30ddfc6df9492e8b6223d6e9ccd912 (127.0.97.1:42661): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0898eb33a4524a4c919663281fe4532e: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42661: connect: Connection refused (error 111)
I20260924 03:13:16.517676 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.498057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.0.97.2:40015)
0924 03:13:16.498272 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:16.498278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:16.498563 (+ 285us) server_negotiation.cc:407] Connection header received
0924 03:13:16.498844 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:16.498851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:16.498942 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:16.499069 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:13:16.500457 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.501680 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.503090 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.503772 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.503898 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:16.510456 (+ 6558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:13:16.510476 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:13:16.510496 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:13:16.510545 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:13:16.514747 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:16.515493 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:16.515502 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:16.515506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:16.515670 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:16.516029 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:16.516034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:16.516038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:16.516456 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:13:16.516812 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:16.517042 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 03:13:16.517274 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:13:16.518460 16792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:13:16.553293 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.552700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42661 (local address 127.0.0.1:51448)
0924 03:13:16.552927 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:16.553037 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 03:13:16.554534 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.554149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42661 (local address 127.0.0.1:51462)
0924 03:13:16.554306 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:16.554390 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 03:13:16.615103 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.614521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42661 (local address 127.0.0.1:51476)
0924 03:13:16.614811 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:16.614917 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260924 03:13:16.616428 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.615984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42661 (local address 127.0.0.1:51480)
0924 03:13:16.616207 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:16.616291 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 03:13:16.635104 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 16669
I20260924 03:13:16.655139 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 16800
I20260924 03:13:16.669433 16940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.655943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41412 (local address 127.0.97.3:38127)
0924 03:13:16.656170 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:16.656174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:16.656192 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:13:16.656342 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:16.656346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:16.656417 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:16.656686 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:13:16.658121 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.658944 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.660341 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:16.660807 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:16.660905 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:16.663615 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:13:16.663629 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:13:16.663644 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:13:16.663676 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:13:16.666555 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:16.667335 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:16.667343 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:16.667345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:16.667492 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:16.667942 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:16.667945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:16.667947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:16.668281 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:13:16.668822 (+ 541us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:16.668967 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 03:13:16.669137 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260924 03:13:16.670236 16923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:13:16.697152 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.696465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42661 (local address 127.0.0.1:51484)
0924 03:13:16.696854 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:16.696965 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 03:13:16.698262 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:16.697828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42661 (local address 127.0.0.1:51488)
0924 03:13:16.698038 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:16.698127 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260924 03:13:16.762454 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 16800
I20260924 03:13:16.779196 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 16447
I20260924 03:13:16.780556 16510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:13:16.897751 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 16447
I20260924 03:13:36.920312 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32789
--webserver_interface=127.0.97.62
--webserver_port=33231
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:13:37.254964 16976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:13:37.255349 16976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:13:37.255528 16976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:13:37.262792 16976 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 03:13:37.262944 16976 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 03:13:37.263038 16976 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 03:13:37.263139 16976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:13:37.263221 16976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:13:37.263338 16976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:13:37.263429 16976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:13:37.263510 16976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:13:37.273758 16976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:13:37.275120 16976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:13:37.276448 16976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:13:37.290493 16982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:37.290481 16981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:13:37.290493 16976 server_base.cc:1034] running on GCE node
W20260924 03:13:37.290558 16984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:13:37.292049 16976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:13:37.293658 16976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:13:37.294919 16976 hybrid_clock.cc:648] HybridClock initialized: now 1790219617294839 us; error 66 us; skew 500 ppm
I20260924 03:13:37.295300 16976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:13:37 dist-test-slave-5l98 krb5kdc[15896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:13:37.303623 16976 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:13:37.305261 16976 webserver.cc:460] Webserver started at http://127.0.97.62:33231/ using document root <none> and password file <none>
I20260924 03:13:37.305938 16976 fs_manager.cc:359] Metadata directory not provided
I20260924 03:13:37.306025 16976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:13:37.312222 16976 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 03:13:37.315196 16991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:37.316805 16976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 03:13:37.316944 16976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "369d40362da64878af413665ae5a3184"
format_stamp: "Formatted at 2026-09-24 03:12:51 on dist-test-slave-5l98"
server_key: "8226170246af57e1628c4e9bbfa0cdcf"
server_key_iv: "2af8407c6c7bf93e30d0cf3f258b7b28"
server_key_version: "encryptionkey@0"
I20260924 03:13:37.317891 16976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:13:37.333386 16976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:13:37.338143 16976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:13:37.338408 16976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:13:37.358969 16976 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32789
I20260924 03:13:37.359146 17042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32789 every 8 connection(s)
I20260924 03:13:37.360684 16976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:13:37.364935 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 16976
I20260924 03:13:37.365787 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42661
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:32789
--webserver_port=45865
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:13:37.368714 17043 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:13:37.372355 17043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184: Bootstrap starting.
I20260924 03:13:37.395192 17043 log.cc:824] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184: Log is configured to *not* fsync() on all Append() calls
I20260924 03:13:37.411478 17043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:13:37.412077 17043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184: Bootstrap complete.
I20260924 03:13:37.421442 17043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369d40362da64878af413665ae5a3184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32789 } }
I20260924 03:13:37.422520 17043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 369d40362da64878af413665ae5a3184, State: Initialized, Role: FOLLOWER
I20260924 03:13:37.423214 17043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369d40362da64878af413665ae5a3184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32789 } }
I20260924 03:13:37.423461 17043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:13:37.423599 17043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:13:37.423900 17043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 2 FOLLOWER]: Advancing to term 3
I20260924 03:13:37.427753 17043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369d40362da64878af413665ae5a3184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32789 } }
I20260924 03:13:37.428249 17043 leader_election.cc:302] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 369d40362da64878af413665ae5a3184; no voters:
I20260924 03:13:37.429037 17043 leader_election.cc:288] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 03:13:37.429188 17047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 03:13:37.430500 17047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [term 3 LEADER]: Becoming Leader. State: Replica: 369d40362da64878af413665ae5a3184, State: Running, Role: LEADER
I20260924 03:13:37.431151 17047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369d40362da64878af413665ae5a3184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32789 } }
I20260924 03:13:37.431324 17043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:13:37.434013 17049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 369d40362da64878af413665ae5a3184. Latest consensus state: current_term: 3 leader_uuid: "369d40362da64878af413665ae5a3184" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369d40362da64878af413665ae5a3184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32789 } } }
I20260924 03:13:37.434414 17049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [sys.catalog]: This master's current role is: LEADER
I20260924 03:13:37.435043 17048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "369d40362da64878af413665ae5a3184" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369d40362da64878af413665ae5a3184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32789 } } }
I20260924 03:13:37.435346 17048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184 [sys.catalog]: This master's current role is: LEADER
I20260924 03:13:37.440517 17055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:13:37.453553 17055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:13:37.455775 17055 catalog_manager.cc:1259] Loaded cluster ID: 08a9c020672f496cb5d322c663646fee
I20260924 03:13:37.455996 17055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:13:37.461745 17055 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:13:37.463979 17055 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184: Loaded TSK: 0
I20260924 03:13:37.464279 17055 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 369d40362da64878af413665ae5a3184: Loaded TSK: 1
F20260924 03:13:37.465912 17055 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbe7c2d19ad google::LogMessage::Fail() at ??:0
@ 0x7fbe7c2d488a google::LogMessage::SendToLog() at ??:0
@ 0x7fbe7c2d14de google::LogMessage::Flush() at ??:0
@ 0x7fbe7c2d2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbe800357d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fbe8002473e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fbe91d001c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fbe91cff783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fbe91d8b2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fbe91d8b0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fbe91dd55e9 std::function<>::operator()() at ??:0
@ 0x7fbe91d03cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fbe91d01c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbe91d89610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fbe91d89439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fbe7ef1d345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbe7ef2091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fbe7ef20799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fbe7eefbeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbe821d56db start_thread at ??:0
@ 0x7fbe7b16b61f clone at ??:0
W20260924 03:13:37.770176 17045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:13:37.770478 17045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:13:37.770653 17045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:13:37.778009 17045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:13:37.778167 17045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:13:37.778366 17045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:13:37.788210 17045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45865
--tserver_master_addrs=127.0.97.62:32789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:13:37.789980 17045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:13:37.791285 17045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:13:37.804618 17091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:37.804728 17090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:37.804670 17093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:13:37.810422 17045 server_base.cc:1034] running on GCE node
I20260924 03:13:37.811096 17045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:13:37.812168 17045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:13:37.813447 17045 hybrid_clock.cc:648] HybridClock initialized: now 1790219617813378 us; error 54 us; skew 500 ppm
I20260924 03:13:37.813871 17045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:13:37 dist-test-slave-5l98 krb5kdc[15896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:13:37.817126 17045 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:13:37.818785 17045 webserver.cc:460] Webserver started at http://127.0.97.1:45865/ using document root <none> and password file <none>
I20260924 03:13:37.819507 17045 fs_manager.cc:359] Metadata directory not provided
I20260924 03:13:37.819610 17045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:13:37.825148 17045 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 03:13:37.828159 17100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:37.829905 17045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:13:37.830090 17045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "cb30ddfc6df9492e8b6223d6e9ccd912"
format_stamp: "Formatted at 2026-09-24 03:12:51 on dist-test-slave-5l98"
server_key: "e50e433e8125fcb44e63e74c921a8eeb"
server_key_iv: "39a56c65beed25b89c173e247772a434"
server_key_version: "encryptionkey@0"
I20260924 03:13:37.831107 17045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:13:37.879370 17045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:13:37.886650 17045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:13:37.887056 17045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:13:37.888481 17045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:13:37.891606 17107 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260924 03:13:37.901556 17107 ts_tablet_manager.cc:1852] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 03:13:37.903671 17107 ts_tablet_manager.cc:1905] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:13:37.915750 17107 ts_tablet_manager.cc:1918] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:13:37.916029 17107 ts_tablet_manager.cc:1939] T 0898eb33a4524a4c919663281fe4532e P cb30ddfc6df9492e8b6223d6e9ccd912: Deleting consensus metadata
I20260924 03:13:37.916472 17045 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 03:13:37.916672 17045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.026s user 0.001s sys 0.000s
I20260924 03:13:37.916842 17045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:13:37.916939 17045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:37.953102 17045 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42661
I20260924 03:13:37.953197 17213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42661 every 8 connection(s)
I20260924 03:13:37.954380 17045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:13:37.959441 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:37.957499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.1:54121)
0924 03:13:37.958325 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:37.958535 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":349,"threads_started":1}
W20260924 03:13:37.960182 17214 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:32789 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:32789: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
I20260924 03:13:37.961043 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:37.960705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.1:43663)
0924 03:13:37.960873 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:37.960941 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 03:13:37.962018 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 17045
I20260924 03:13:37.962056 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:37.961731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.1:42569)
0924 03:13:37.961869 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:37.961943 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 03:13:37.962320 17214 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:13:37.962729 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40015
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:32789
--webserver_port=43479
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:13:38.297894 17217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:13:38.298254 17217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:13:38.298429 17217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:13:38.306188 17217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:13:38.306367 17217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:13:38.306548 17217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:13:38.316746 17217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43479
--tserver_master_addrs=127.0.97.62:32789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:13:38.318125 17217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:13:38.319507 17217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:13:38.333559 17225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:38.333613 17223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:38.333660 17222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:13:38.334296 17217 server_base.cc:1034] running on GCE node
I20260924 03:13:38.334970 17217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:13:38.336078 17217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:13:38.337317 17217 hybrid_clock.cc:648] HybridClock initialized: now 1790219618337278 us; error 37 us; skew 500 ppm
I20260924 03:13:38.337700 17217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:13:38 dist-test-slave-5l98 krb5kdc[15896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:13:38.341126 17217 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:13:38.342602 17217 webserver.cc:460] Webserver started at http://127.0.97.2:43479/ using document root <none> and password file <none>
I20260924 03:13:38.343304 17217 fs_manager.cc:359] Metadata directory not provided
I20260924 03:13:38.343463 17217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:13:38.349480 17217 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 03:13:38.352661 17232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:38.354449 17217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 03:13:38.354668 17217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "9b765b6326824826a0d2e92c7859ac00"
format_stamp: "Formatted at 2026-09-24 03:12:52 on dist-test-slave-5l98"
server_key: "9a54bed913ba8bc2dbf23a8945558336"
server_key_iv: "864c70f9bdc29919e6f2a3f047da0643"
server_key_version: "encryptionkey@0"
I20260924 03:13:38.355742 17217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:13:38.371107 17217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:13:38.375772 17217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:13:38.376084 17217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:13:38.377219 17217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:13:38.378585 17217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:13:38.378672 17217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:38.378784 17217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:13:38.378865 17217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:38.415148 17217 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40015
I20260924 03:13:38.415191 17344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40015 every 8 connection(s)
I20260924 03:13:38.416715 17217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:13:38.417454 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 17217
I20260924 03:13:38.418394 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:38127
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:32789
--webserver_port=45603
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:13:38.422605 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:38.420286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.2:52107)
0924 03:13:38.421392 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:38.421656 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":562,"threads_started":1}
W20260924 03:13:38.423416 17345 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:32789 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:32789: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
I20260924 03:13:38.424333 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:38.423984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.2:56333)
0924 03:13:38.424149 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:38.424228 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 03:13:38.425259 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:38.424880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.2:49885)
0924 03:13:38.425020 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:38.425107 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 03:13:38.425595 17345 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:13:38.750622 17348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:13:38.750924 17348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:13:38.751027 17348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:13:38.758636 17348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:13:38.758780 17348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:13:38.758910 17348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:13:38.769375 17348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:38127
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45603
--tserver_master_addrs=127.0.97.62:32789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:13:38.770687 17348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:13:38.771934 17348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:13:38.785805 17354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:38.785909 17356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:38.785835 17353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:13:38.787230 17348 server_base.cc:1034] running on GCE node
I20260924 03:13:38.787990 17348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:13:38.789160 17348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:13:38.790422 17348 hybrid_clock.cc:648] HybridClock initialized: now 1790219618790369 us; error 46 us; skew 500 ppm
I20260924 03:13:38.790959 17348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:13:38 dist-test-slave-5l98 krb5kdc[15896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:13:38.794653 17348 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:13:38.796087 17348 webserver.cc:460] Webserver started at http://127.0.97.3:45603/ using document root <none> and password file <none>
I20260924 03:13:38.796835 17348 fs_manager.cc:359] Metadata directory not provided
I20260924 03:13:38.796945 17348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:13:38.802884 17348 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 03:13:38.805902 17363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:38.807375 17348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 03:13:38.807588 17348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "7371cf39fb024460af08c87a3e1269d9"
format_stamp: "Formatted at 2026-09-24 03:12:52 on dist-test-slave-5l98"
server_key: "d8337a2a1868243069a8139c031e4ad0"
server_key_iv: "fa40babc994cf5fda7c4d13f1ccc3593"
server_key_version: "encryptionkey@0"
I20260924 03:13:38.808890 17348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:13:38.823406 17348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:13:38.828270 17348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:13:38.828611 17348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:13:38.829689 17348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:13:38.831162 17348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:13:38.831295 17348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:38.831440 17348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:13:38.831538 17348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:38.870736 17348 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38127
I20260924 03:13:38.870784 17475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38127 every 8 connection(s)
I20260924 03:13:38.872334 17348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:13:38.874233 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 17348
I20260924 03:13:38.878176 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:38.876141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.3:53281)
0924 03:13:38.877013 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:38.877219 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":418,"threads_started":1}
W20260924 03:13:38.878882 17476 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:32789 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:32789: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
I20260924 03:13:38.879868 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:38.879492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.3:34695)
0924 03:13:38.879663 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:38.879751 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 03:13:38.880764 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:38.880370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.3:55519)
0924 03:13:38.880555 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:38.880641 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 03:13:38.880988 17476 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:13:38.963804 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:38.962939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.1:49367)
0924 03:13:38.963453 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:38.963540 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":271,"threads_started":1}
I20260924 03:13:39.427256 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:39.426273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.2:53731)
0924 03:13:39.426887 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:39.427005 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":255,"threads_started":1}
I20260924 03:13:39.882529 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:39.881608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.3:36509)
0924 03:13:39.882156 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:39.882269 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":273,"threads_started":1}
I20260924 03:13:39.965680 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:39.964904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.1:36167)
0924 03:13:39.965351 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:39.965453 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":232,"threads_started":1}
I20260924 03:13:40.429316 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:40.428414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.2:54771)
0924 03:13:40.428991 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:40.429098 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":334,"threads_started":1}
I20260924 03:13:40.884711 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:40.883807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.3:38119)
0924 03:13:40.884299 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:40.884393 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":236,"threads_started":1}
I20260924 03:13:40.967439 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:40.966724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.1:40147)
0924 03:13:40.967194 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:40.967283 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":236,"threads_started":1}
I20260924 03:13:41.431027 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:41.430234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.2:40829)
0924 03:13:41.430750 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:41.430848 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":258,"threads_started":1}
I20260924 03:13:41.886565 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:41.885767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.3:47845)
0924 03:13:41.886255 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:41.886374 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":255,"threads_started":1}
I20260924 03:13:41.969363 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:41.968451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.1:48809)
0924 03:13:41.969091 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:41.969175 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":351,"threads_started":1}
I20260924 03:13:42.433208 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:42.432256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.2:34117)
0924 03:13:42.432924 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:42.433014 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":458,"threads_started":1}
I20260924 03:13:42.888830 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:42.887873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.3:43971)
0924 03:13:42.888389 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:42.888470 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":252,"threads_started":1}
I20260924 03:13:42.971204 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:42.970408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.1:41007)
0924 03:13:42.970960 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:42.971043 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":321,"threads_started":1}
I20260924 03:13:43.435146 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:43.434220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.2:45943)
0924 03:13:43.434868 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:43.434966 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":274,"threads_started":1}
I20260924 03:13:43.890784 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:43.890003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.3:38941)
0924 03:13:43.890509 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:43.890599 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":273,"threads_started":1}
I20260924 03:13:43.973235 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:43.972422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.1:49171)
0924 03:13:43.972977 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:43.973058 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":264,"threads_started":1}
I20260924 03:13:44.437261 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:44.436317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.2:51969)
0924 03:13:44.436924 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:44.437015 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":363,"threads_started":1}
I20260924 03:13:44.892742 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:44.891876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.3:43139)
0924 03:13:44.892390 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:44.892549 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":280,"threads_started":1}
I20260924 03:13:44.974972 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:44.974151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.1:40659)
0924 03:13:44.974670 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:44.974764 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":248,"threads_started":1}
I20260924 03:13:45.439437 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:45.438300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.2:56699)
0924 03:13:45.439118 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:45.439216 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":450,"threads_started":1}
I20260924 03:13:45.894563 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:45.893716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.3:46781)
0924 03:13:45.894267 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:45.894354 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":258,"threads_started":1}
I20260924 03:13:45.976841 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:45.976066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.1:54731)
0924 03:13:45.976557 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:45.976641 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":255,"threads_started":1}
I20260924 03:13:46.441099 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:46.440362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.2:51917)
0924 03:13:46.440854 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:46.440937 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":267,"threads_started":1}
I20260924 03:13:46.896687 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:46.895726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.3:39183)
0924 03:13:46.896321 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:46.896418 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":298,"threads_started":1}
I20260924 03:13:46.978715 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:46.977865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.1:56129)
0924 03:13:46.978359 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:46.978450 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":256,"threads_started":1}
I20260924 03:13:47.442926 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:47.442140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.2:43603)
0924 03:13:47.442654 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:47.442742 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":266,"threads_started":1}
I20260924 03:13:47.898933 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:47.897886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.3:56325)
0924 03:13:47.898478 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:47.898650 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":290,"threads_started":1}
I20260924 03:13:47.980441 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:47.979731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.1:51287)
0924 03:13:47.980170 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:47.980261 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":235,"threads_started":1}
I20260924 03:13:48.444940 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:48.444066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.2:56261)
0924 03:13:48.444623 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:48.444706 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":251,"threads_started":1}
I20260924 03:13:48.900841 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:48.899993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.3:43561)
0924 03:13:48.900550 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:48.900660 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":284,"threads_started":1}
I20260924 03:13:48.982412 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:48.981562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.1:48093)
0924 03:13:48.982137 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:48.982222 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":327,"threads_started":1}
I20260924 03:13:49.446581 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:49.445843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.2:55205)
0924 03:13:49.446309 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:49.446399 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":239,"threads_started":1}
I20260924 03:13:49.902817 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:49.902008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.3:40659)
0924 03:13:49.902507 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:49.902610 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":265,"threads_started":1}
I20260924 03:13:49.984452 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:49.983506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.1:48685)
0924 03:13:49.984184 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:49.984265 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":452,"threads_started":1}
I20260924 03:13:50.448712 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:50.447817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.2:48139)
0924 03:13:50.448341 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:50.448430 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":290,"threads_started":1}
I20260924 03:13:50.905165 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:50.904162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.3:55731)
0924 03:13:50.904892 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:50.904993 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":346,"threads_started":1}
I20260924 03:13:50.986266 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:50.985469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.1:59275)
0924 03:13:50.985995 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:50.986075 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":307,"threads_started":1}
I20260924 03:13:51.450585 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:51.449793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.2:48489)
0924 03:13:51.450288 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:51.450389 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":233,"threads_started":1}
I20260924 03:13:51.906881 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:51.906117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.3:40397)
0924 03:13:51.906598 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:51.906687 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":262,"threads_started":1}
I20260924 03:13:51.987986 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:51.987252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.1:47379)
0924 03:13:51.987709 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:51.987795 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":241,"threads_started":1}
I20260924 03:13:52.452363 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:52.451588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.2:38949)
0924 03:13:52.452049 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:52.452176 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":240,"threads_started":1}
I20260924 03:13:52.908838 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:52.907870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.3:46553)
0924 03:13:52.908442 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:52.908563 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":269,"threads_started":1}
I20260924 03:13:52.990199 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:52.989385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.1:50473)
0924 03:13:52.989915 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:52.990003 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":273,"threads_started":1}
I20260924 03:13:53.454445 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:53.453659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.2:45853)
0924 03:13:53.454153 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:53.454238 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":247,"threads_started":1}
I20260924 03:13:53.883991 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 17045
I20260924 03:13:53.910759 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:53.909989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32789 (local address 127.0.97.3:52685)
0924 03:13:53.910511 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:53.910588 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":270,"threads_started":1}
I20260924 03:13:53.911731 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:53.885356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51694 (local address 127.0.97.1:42661)
0924 03:13:53.886045 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:53.886065 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:53.886150 (+ 85us) server_negotiation.cc:407] Connection header received
0924 03:13:53.888002 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:53.888011 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:53.888440 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:53.888868 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:13:53.890396 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:53.891626 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:53.892975 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:53.893647 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:53.893842 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:53.904726 (+ 10884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:13:53.904735 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:13:53.904754 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:13:53.904802 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:13:53.908132 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:53.909037 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:53.909045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:53.909052 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:53.909228 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:53.909662 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:53.909666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:53.909668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:53.910237 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:13:53.910568 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:53.911014 (+ 446us) server_negotiation.cc:299] Negotiation successful
0924 03:13:53.911448 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":317,"threads_started":1}
I20260924 03:13:53.913316 17209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:13:53.999675 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 17045
I20260924 03:13:54.017351 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 17217
I20260924 03:13:54.036679 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:54.018371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40464 (local address 127.0.97.2:40015)
0924 03:13:54.018981 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:54.018996 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:54.019033 (+ 37us) server_negotiation.cc:407] Connection header received
0924 03:13:54.020653 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:54.020664 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:54.021072 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:54.021370 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:13:54.022765 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:54.024192 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:54.025672 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:54.026367 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:54.026506 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:54.029397 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:13:54.029405 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:13:54.029421 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:13:54.029469 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:13:54.033108 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:54.033935 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:54.033943 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:54.033949 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:54.034118 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:54.034559 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:54.034566 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:54.034569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:54.035107 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:13:54.035430 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:54.035841 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 03:13:54.036268 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":324,"threads_started":1}
I20260924 03:13:54.038035 17340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:13:54.133234 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 17217
I20260924 03:13:54.152033 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 17348
I20260924 03:13:54.172428 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:54.153436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55252 (local address 127.0.97.3:38127)
0924 03:13:54.154354 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:54.154372 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:54.154425 (+ 53us) server_negotiation.cc:407] Connection header received
0924 03:13:54.156740 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:54.156751 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:54.157187 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:54.157551 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:13:54.159052 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:54.160267 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:54.161579 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:54.162267 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:54.162467 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:54.165288 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:13:54.165295 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:13:54.165311 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:13:54.165346 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:13:54.168614 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:54.169500 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:54.169510 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:54.169522 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:54.169737 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:54.170170 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:54.170178 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:54.170181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:54.170774 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:13:54.171119 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:54.171582 (+ 463us) server_negotiation.cc:299] Negotiation successful
0924 03:13:54.172061 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":460,"threads_started":1}
I20260924 03:13:54.173921 17471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:13:54.264209 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 17348
2026-09-24T03:13:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64666 ms)
[----------] 30 tests from SecurityITest (307058 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:13:54 dist-test-slave-5l98 krb5kdc[17534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:13:54 dist-test-slave-5l98 krb5kdc[17534](info): set up 2 sockets
Sep 24 03:13:54 dist-test-slave-5l98 krb5kdc[17534](info): commencing operation
krb5kdc: starting...
W20260924 03:13:55.416945 388 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:13:55 dist-test-slave-5l98 krb5kdc[17534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:13:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:13:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:13:55.685877 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39403
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:13:56.025919 17547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:13:56.026221 17547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:13:56.026436 17547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:13:56.033931 17547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:13:56.034052 17547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:13:56.034152 17547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:13:56.034268 17547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:13:56.034336 17547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:13:56.044811 17547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:13:56.046380 17547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:13:56.047744 17547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:13:56.060056 17552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:56.060164 17553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:56.060122 17555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:13:56.061359 17547 server_base.cc:1034] running on GCE node
I20260924 03:13:56.062187 17547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:13:56.063736 17547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:13:56.064971 17547 hybrid_clock.cc:648] HybridClock initialized: now 1790219636064937 us; error 54 us; skew 500 ppm
I20260924 03:13:56.065337 17547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:13:56 dist-test-slave-5l98 krb5kdc[17534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:13:56.068675 17547 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:13:56.070230 17547 webserver.cc:460] Webserver started at http://127.0.97.62:38249/ using document root <none> and password file <none>
I20260924 03:13:56.070935 17547 fs_manager.cc:359] Metadata directory not provided
I20260924 03:13:56.071085 17547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:13:56.071404 17547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:13:56.074488 17547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "861bf5cc313440ab84d38812c88e988f"
format_stamp: "Formatted at 2026-09-24 03:13:56 on dist-test-slave-5l98"
server_key: "85a50a30cd43b2e817043d237b9501ff"
server_key_iv: "826d6f7d709c8d536945acd7b92c7bc9"
server_key_version: "encryptionkey@0"
I20260924 03:13:56.075254 17547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "861bf5cc313440ab84d38812c88e988f"
format_stamp: "Formatted at 2026-09-24 03:13:56 on dist-test-slave-5l98"
server_key: "85a50a30cd43b2e817043d237b9501ff"
server_key_iv: "826d6f7d709c8d536945acd7b92c7bc9"
server_key_version: "encryptionkey@0"
I20260924 03:13:56.080376 17547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:13:56.084038 17562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:56.085700 17547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 03:13:56.085920 17547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "861bf5cc313440ab84d38812c88e988f"
format_stamp: "Formatted at 2026-09-24 03:13:56 on dist-test-slave-5l98"
server_key: "85a50a30cd43b2e817043d237b9501ff"
server_key_iv: "826d6f7d709c8d536945acd7b92c7bc9"
server_key_version: "encryptionkey@0"
I20260924 03:13:56.086141 17547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:13:56.113312 17547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:13:56.118112 17547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:13:56.118439 17547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:13:56.137912 17547 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39403
I20260924 03:13:56.137950 17613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39403 every 8 connection(s)
I20260924 03:13:56.139432 17547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:13:56.142069 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 17547
I20260924 03:13:56.142295 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:13:56.143185 388 external_mini_cluster.cc:1428] Setting key af8f201ae76998c23d2e170951bf2bd5
I20260924 03:13:56.144389 17614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 03:13:56 dist-test-slave-5l98 krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:13:56.159844 17614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f: Bootstrap starting.
I20260924 03:13:56.164090 17614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f: Neither blocks nor log segments found. Creating new log.
I20260924 03:13:56.165396 17614 log.cc:824] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:13:56.168908 17614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f: No bootstrap required, opened a new log
I20260924 03:13:56.169348 17617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:56.145522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42658 (local address 127.0.97.62:39403)
0924 03:13:56.146347 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:56.146359 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:56.146398 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:13:56.147986 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:56.148006 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:56.148423 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:56.148852 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:13:56.150844 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:56.152297 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:56.154302 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:56.155022 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:56.155164 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:56.160287 (+ 5123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:13:56.160295 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:13:56.160311 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:13:56.160381 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:13:56.164720 (+ 4339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:56.165545 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:56.165558 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:56.165567 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:56.165789 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:56.166250 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:56.166254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:56.166256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:56.166845 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:13:56.167137 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:56.167525 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 03:13:56.167972 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":381,"threads_started":1}
I20260924 03:13:56.177682 17614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861bf5cc313440ab84d38812c88e988f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39403 } }
I20260924 03:13:56.178251 17614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:13:56.178372 17614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861bf5cc313440ab84d38812c88e988f, State: Initialized, Role: FOLLOWER
I20260924 03:13:56.179073 17614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861bf5cc313440ab84d38812c88e988f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39403 } }
I20260924 03:13:56.179327 17614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:13:56.179482 17614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:13:56.179656 17614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:13:56.182036 17614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861bf5cc313440ab84d38812c88e988f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39403 } }
I20260924 03:13:56.182651 17614 leader_election.cc:302] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 861bf5cc313440ab84d38812c88e988f; no voters:
I20260924 03:13:56.183233 17614 leader_election.cc:288] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:13:56.183384 17619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:13:56.184118 17619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f [term 1 LEADER]: Becoming Leader. State: Replica: 861bf5cc313440ab84d38812c88e988f, State: Running, Role: LEADER
I20260924 03:13:56.184819 17619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861bf5cc313440ab84d38812c88e988f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39403 } }
I20260924 03:13:56.185470 17614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:13:56.187978 17621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 861bf5cc313440ab84d38812c88e988f. Latest consensus state: current_term: 1 leader_uuid: "861bf5cc313440ab84d38812c88e988f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861bf5cc313440ab84d38812c88e988f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39403 } } }
I20260924 03:13:56.187973 17620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "861bf5cc313440ab84d38812c88e988f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861bf5cc313440ab84d38812c88e988f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39403 } } }
I20260924 03:13:56.188406 17621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f [sys.catalog]: This master's current role is: LEADER
I20260924 03:13:56.188448 17620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f [sys.catalog]: This master's current role is: LEADER
I20260924 03:13:56.192348 17626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:13:56.207056 17626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:13:56.219092 17626 catalog_manager.cc:1347] Generated new cluster ID: 46eb1d76ee1c4c61862c7d4db94d35ab
I20260924 03:13:56.219321 17626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:13:56.248623 17626 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:13:56.250734 17626 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:13:56.270265 17626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f: Generated new TSK 0
I20260924 03:13:56.271364 17626 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 03:13:56.337000 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39403
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:13:56.691263 17640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:13:56.691555 17640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:13:56.691735 17640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:13:56.699327 17640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:13:56.699460 17640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:13:56.699652 17640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:13:56.709723 17640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:13:56.711045 17640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:13:56.712330 17640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:13:56.730515 17645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:56.730496 17648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:56.730593 17646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:13:56.731539 17640 server_base.cc:1034] running on GCE node
I20260924 03:13:56.732263 17640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:13:56.733381 17640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:13:56.734737 17640 hybrid_clock.cc:648] HybridClock initialized: now 1790219636734532 us; error 95 us; skew 500 ppm
I20260924 03:13:56.735165 17640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:13:56 dist-test-slave-5l98 krb5kdc[17534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:13:56.739023 17640 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:13:56.740720 17640 webserver.cc:460] Webserver started at http://127.0.97.1:32963/ using document root <none> and password file <none>
I20260924 03:13:56.741495 17640 fs_manager.cc:359] Metadata directory not provided
I20260924 03:13:56.741597 17640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:13:56.741861 17640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:13:56.744767 17640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "e6fc21553ffa411497a89666805faf99"
format_stamp: "Formatted at 2026-09-24 03:13:56 on dist-test-slave-5l98"
server_key: "fcf8f794a014ec3e8fc19b4ea4f79fc4"
server_key_iv: "5dc7e85be6e612506f8b481315e25bb1"
server_key_version: "encryptionkey@0"
I20260924 03:13:56.745501 17640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e6fc21553ffa411497a89666805faf99"
format_stamp: "Formatted at 2026-09-24 03:13:56 on dist-test-slave-5l98"
server_key: "fcf8f794a014ec3e8fc19b4ea4f79fc4"
server_key_iv: "5dc7e85be6e612506f8b481315e25bb1"
server_key_version: "encryptionkey@0"
I20260924 03:13:56.750783 17640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 03:13:56.754544 17655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:56.755913 17640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:13:56.756111 17640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "e6fc21553ffa411497a89666805faf99"
format_stamp: "Formatted at 2026-09-24 03:13:56 on dist-test-slave-5l98"
server_key: "fcf8f794a014ec3e8fc19b4ea4f79fc4"
server_key_iv: "5dc7e85be6e612506f8b481315e25bb1"
server_key_version: "encryptionkey@0"
I20260924 03:13:56.756328 17640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:13:56.782333 17640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:13:56.787202 17640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:13:56.787515 17640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:13:56.788575 17640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:13:56.790060 17640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:13:56.790174 17640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:56.790320 17640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:13:56.790392 17640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:56.825062 17640 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37367
I20260924 03:13:56.825119 17767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37367 every 8 connection(s)
I20260924 03:13:56.826314 17640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:13:56.833408 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 17640
I20260924 03:13:56.833621 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:13:56.834421 388 external_mini_cluster.cc:1428] Setting key d6d2ddbe8a3ec614a5ebb1648eddb5ee
Sep 24 03:13:56 dist-test-slave-5l98 krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:13:56.847888 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:56.828980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52867 (local address 127.0.97.62:39403)
0924 03:13:56.829455 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:56.829462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:56.830262 (+ 800us) server_negotiation.cc:407] Connection header received
0924 03:13:56.832118 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:56.832123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:56.832202 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:56.832319 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:13:56.834011 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:56.835096 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:56.836763 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:56.837149 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:56.837303 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:56.840927 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:13:56.840937 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:13:56.840945 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:13:56.840994 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:13:56.844440 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:56.845573 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:56.845577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:56.845579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:56.845694 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:56.846183 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:56.846186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:56.846188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:56.846461 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:13:56.846573 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:56.847459 (+ 886us) server_negotiation.cc:299] Negotiation successful
0924 03:13:56.847632 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":225,"threads_started":1}
I20260924 03:13:56.849455 17771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:56.829324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39403 (local address 127.0.97.1:52867)
0924 03:13:56.830086 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:56.830133 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 03:13:56.831829 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:13:56.832607 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:13:56.832615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:13:56.833147 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:13:56.833834 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:56.833853 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:56.835279 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:56.835284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:56.836610 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:56.836628 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:56.837392 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:56.837401 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:56.837638 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:56.838356 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:13:56.838379 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:13:56.840610 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:13:56.844693 (+ 4083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:13:56.844738 (+ 45us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:13:56.844758 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:13:56.845434 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:13:56.845809 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:13:56.845817 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:13:56.845823 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:13:56.846064 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:13:56.846590 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:13:56.846600 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:13:56.847058 (+ 458us) client_negotiation.cc:769] Sending connection context
0924 03:13:56.847351 (+ 293us) client_negotiation.cc:240] Negotiation successful
0924 03:13:56.847949 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":316,"threads_started":1}
I20260924 03:13:56.851118 17768 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39403
I20260924 03:13:56.851467 17768 heartbeater.cc:464] Registering TS with master...
I20260924 03:13:56.852293 17768 heartbeater.cc:511] Master 127.0.97.62:39403 requested a full tablet report, sending...
I20260924 03:13:56.854569 17579 ts_manager.cc:194] Registered new tserver with Master: e6fc21553ffa411497a89666805faf99 (127.0.97.1:37367)
I20260924 03:13:56.856226 17579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52867
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 03:13:56.899258 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39403
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:13:57.245903 17774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:13:57.246188 17774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:13:57.246297 17774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:13:57.253962 17774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:13:57.254077 17774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:13:57.254192 17774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:13:57.264576 17774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:13:57.265815 17774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:13:57.267078 17774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:13:57.280560 17782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:57.280539 17780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:57.280755 17779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:13:57.281466 17774 server_base.cc:1034] running on GCE node
I20260924 03:13:57.282032 17774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:13:57.283051 17774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:13:57.284380 17774 hybrid_clock.cc:648] HybridClock initialized: now 1790219637284324 us; error 57 us; skew 500 ppm
I20260924 03:13:57.284775 17774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:13:57 dist-test-slave-5l98 krb5kdc[17534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:13:57.288173 17774 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:13:57.289983 17774 webserver.cc:460] Webserver started at http://127.0.97.2:34229/ using document root <none> and password file <none>
I20260924 03:13:57.290685 17774 fs_manager.cc:359] Metadata directory not provided
I20260924 03:13:57.290853 17774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:13:57.291162 17774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:13:57.294108 17774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "11e76ed6fa3d4812ba5369c27cbf55a1"
format_stamp: "Formatted at 2026-09-24 03:13:57 on dist-test-slave-5l98"
server_key: "69944f902492209d9d044866ef73f3f4"
server_key_iv: "17cfd7b4cdf6f7d852fd7352735b0c15"
server_key_version: "encryptionkey@0"
I20260924 03:13:57.294983 17774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "11e76ed6fa3d4812ba5369c27cbf55a1"
format_stamp: "Formatted at 2026-09-24 03:13:57 on dist-test-slave-5l98"
server_key: "69944f902492209d9d044866ef73f3f4"
server_key_iv: "17cfd7b4cdf6f7d852fd7352735b0c15"
server_key_version: "encryptionkey@0"
I20260924 03:13:57.300099 17774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:13:57.303620 17789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:57.305334 17774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:13:57.305516 17774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "11e76ed6fa3d4812ba5369c27cbf55a1"
format_stamp: "Formatted at 2026-09-24 03:13:57 on dist-test-slave-5l98"
server_key: "69944f902492209d9d044866ef73f3f4"
server_key_iv: "17cfd7b4cdf6f7d852fd7352735b0c15"
server_key_version: "encryptionkey@0"
I20260924 03:13:57.305724 17774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:13:57.322290 17774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:13:57.326884 17774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:13:57.327206 17774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:13:57.328292 17774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:13:57.329753 17774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:13:57.329893 17774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:57.330049 17774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:13:57.330127 17774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:57.366806 17774 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35233
I20260924 03:13:57.366871 17901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35233 every 8 connection(s)
I20260924 03:13:57.368129 17774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:13:57.376163 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 17774
I20260924 03:13:57.376441 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:13:57.377002 388 external_mini_cluster.cc:1428] Setting key 43be65ba0eb80ab7b72e624cc559d9de
Sep 24 03:13:57 dist-test-slave-5l98 krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:13:57.392655 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:57.371048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33941 (local address 127.0.97.62:39403)
0924 03:13:57.371541 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:57.371549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:57.372623 (+ 1074us) server_negotiation.cc:407] Connection header received
0924 03:13:57.374708 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:57.374713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:57.374790 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:57.374899 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:13:57.376528 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:57.377236 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:57.379025 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:57.379869 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:57.380021 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:57.384152 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:13:57.384162 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:13:57.384170 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:13:57.384223 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:13:57.388563 (+ 4340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:57.389817 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:57.389821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:57.389824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:57.390006 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:57.390577 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:57.390582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:57.390584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:57.391048 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:13:57.391238 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:57.391987 (+ 749us) server_negotiation.cc:299] Negotiation successful
0924 03:13:57.392225 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":236,"threads_started":1}
I20260924 03:13:57.393630 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:57.371656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39403 (local address 127.0.97.2:33941)
0924 03:13:57.372403 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:57.372445 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:13:57.374341 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:13:57.375144 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:13:57.375154 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:13:57.375678 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:13:57.376311 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:57.376341 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:57.377400 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:57.377405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:57.378826 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:57.378839 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:57.380111 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:57.380116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:57.380356 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:57.381173 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:13:57.381197 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:13:57.383873 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:13:57.388766 (+ 4893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:13:57.388778 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:13:57.388801 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:13:57.389520 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:13:57.390183 (+ 663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:13:57.390190 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:13:57.390197 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:13:57.390453 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:13:57.391252 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:13:57.391263 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:13:57.391618 (+ 355us) client_negotiation.cc:769] Sending connection context
0924 03:13:57.391897 (+ 279us) client_negotiation.cc:240] Negotiation successful
0924 03:13:57.392481 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":320,"threads_started":1}
I20260924 03:13:57.395159 17902 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39403
I20260924 03:13:57.395493 17902 heartbeater.cc:464] Registering TS with master...
I20260924 03:13:57.396227 17902 heartbeater.cc:511] Master 127.0.97.62:39403 requested a full tablet report, sending...
I20260924 03:13:57.397943 17579 ts_manager.cc:194] Registered new tserver with Master: 11e76ed6fa3d4812ba5369c27cbf55a1 (127.0.97.2:35233)
I20260924 03:13:57.398802 17579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33941
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 03:13:57.440723 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39403
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:13:57.779084 17908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:13:57.779408 17908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:13:57.779507 17908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:13:57.787331 17908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:13:57.787460 17908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:13:57.787599 17908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:13:57.797996 17908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:13:57.799374 17908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:13:57.800711 17908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:13:57.815412 17914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:57.815601 17913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:57.815745 17916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:13:57.822044 17908 server_base.cc:1034] running on GCE node
I20260924 03:13:57.822790 17908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:13:57.823947 17908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:13:57.825234 17908 hybrid_clock.cc:648] HybridClock initialized: now 1790219637825186 us; error 37 us; skew 500 ppm
I20260924 03:13:57.825599 17908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:13:57 dist-test-slave-5l98 krb5kdc[17534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:13:57.829157 17908 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:13:57.830928 17908 webserver.cc:460] Webserver started at http://127.0.97.3:42649/ using document root <none> and password file <none>
I20260924 03:13:57.831666 17908 fs_manager.cc:359] Metadata directory not provided
I20260924 03:13:57.831781 17908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:13:57.832104 17908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:13:57.835222 17908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "fda50413e68d4d5daaadc487da03e9bb"
format_stamp: "Formatted at 2026-09-24 03:13:57 on dist-test-slave-5l98"
server_key: "407b75dd9597fb681950d68774dcc1cc"
server_key_iv: "79174f95c930965ddea7359b5cebc26d"
server_key_version: "encryptionkey@0"
I20260924 03:13:57.836035 17908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fda50413e68d4d5daaadc487da03e9bb"
format_stamp: "Formatted at 2026-09-24 03:13:57 on dist-test-slave-5l98"
server_key: "407b75dd9597fb681950d68774dcc1cc"
server_key_iv: "79174f95c930965ddea7359b5cebc26d"
server_key_version: "encryptionkey@0"
I20260924 03:13:57.841388 17908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 03:13:57.845050 17923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:57.846506 17908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 03:13:57.846704 17908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "fda50413e68d4d5daaadc487da03e9bb"
format_stamp: "Formatted at 2026-09-24 03:13:57 on dist-test-slave-5l98"
server_key: "407b75dd9597fb681950d68774dcc1cc"
server_key_iv: "79174f95c930965ddea7359b5cebc26d"
server_key_version: "encryptionkey@0"
I20260924 03:13:57.846935 17908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:13:57.859591 17768 heartbeater.cc:503] Master 127.0.97.62:39403 was elected leader, sending a full tablet report...
I20260924 03:13:57.863251 17908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:13:57.868088 17908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:13:57.868417 17908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:13:57.869592 17908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:13:57.870991 17908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:13:57.871061 17908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:57.871189 17908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:13:57.871241 17908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:57.905845 17908 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37983
I20260924 03:13:57.905964 18035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37983 every 8 connection(s)
I20260924 03:13:57.907146 17908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:13:57.916523 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 17908
I20260924 03:13:57.916715 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:13:57.917281 388 external_mini_cluster.cc:1428] Setting key 6a515ff7bfbdd142337afcad5ef6ebe6
Sep 24 03:13:57 dist-test-slave-5l98 krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:13:57.927901 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:57.910070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45135 (local address 127.0.97.62:39403)
0924 03:13:57.910600 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:57.910605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:57.911465 (+ 860us) server_negotiation.cc:407] Connection header received
0924 03:13:57.913406 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:57.913412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:57.913511 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:57.913634 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:13:57.915084 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:57.915847 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:57.918151 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:57.918710 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:57.918838 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:57.922530 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:13:57.922534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:13:57.922538 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:13:57.922570 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:13:57.924988 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:57.925694 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:57.925698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:57.925700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:57.925809 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:57.926330 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:57.926334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:57.926336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:57.926595 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:13:57.926721 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:57.927474 (+ 753us) server_negotiation.cc:299] Negotiation successful
0924 03:13:57.927640 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":204,"threads_started":1}
I20260924 03:13:57.928956 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:57.910572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39403 (local address 127.0.97.3:45135)
0924 03:13:57.911302 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:57.911343 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:13:57.913142 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:13:57.913860 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:13:57.913868 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:13:57.914375 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:13:57.914915 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:57.914931 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:57.916001 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:57.916008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:57.918004 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:57.918016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:57.918965 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:57.918972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:57.919171 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:57.919946 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:13:57.919977 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:13:57.922321 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:13:57.925118 (+ 2797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:13:57.925125 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:13:57.925138 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:13:57.925572 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:13:57.925942 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:13:57.925945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:13:57.925948 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:13:57.926161 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:13:57.926719 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:13:57.926727 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:13:57.927074 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 03:13:57.927370 (+ 296us) client_negotiation.cc:240] Negotiation successful
0924 03:13:57.927753 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":323,"threads_started":1}
I20260924 03:13:57.930339 18036 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39403
I20260924 03:13:57.930680 18036 heartbeater.cc:464] Registering TS with master...
I20260924 03:13:57.931528 18036 heartbeater.cc:511] Master 127.0.97.62:39403 requested a full tablet report, sending...
I20260924 03:13:57.933135 17579 ts_manager.cc:194] Registered new tserver with Master: fda50413e68d4d5daaadc487da03e9bb (127.0.97.3:37983)
I20260924 03:13:57.933934 17579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45135
I20260924 03:13:57.945252 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:13:57.963574 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:57.949990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.0.97.62:39403)
0924 03:13:57.950279 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:57.950289 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:57.950645 (+ 356us) server_negotiation.cc:407] Connection header received
0924 03:13:57.950942 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:57.950949 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:57.951021 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:57.951107 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:13:57.952739 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:57.953461 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:57.954933 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:57.955368 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:57.955467 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:57.957918 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:13:57.957923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:13:57.957928 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:13:57.957961 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:13:57.960845 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:57.961646 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:57.961650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:57.961652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:57.961780 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:57.962206 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:57.962209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:57.962211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:57.962550 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:13:57.962747 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:57.963045 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 03:13:57.963218 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 03:13:57.968686 17579 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:13:57.979568 17579 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:13:58.010468 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:57.996289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.0.1:50318)
0924 03:13:57.997669 (+ 1380us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:57.997725 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 03:13:57.997932 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:13:57.999767 (+ 1835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:13:57.999778 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:13:57.999832 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:13:58.000279 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:58.000294 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.002022 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.002030 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:58.005070 (+ 3040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:58.005095 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.007850 (+ 2755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.007860 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:58.008161 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:58.008238 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 03:13:58.008435 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 03:13:58.009878 (+ 1443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1081,"thread_start_us":12216,"threads_started":1}
I20260924 03:13:58.011406 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:57.997656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50318 (local address 127.0.97.1:37367)
0924 03:13:57.998282 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:57.998288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:57.998316 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:13:57.998391 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:57.998396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:57.998674 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:57.998865 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:13:58.000443 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.001824 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.005340 (+ 3516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.007170 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.007315 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:58.007652 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:58.010770 (+ 3118us) server_negotiation.cc:299] Negotiation successful
0924 03:13:58.011028 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":803,"threads_started":1}
I20260924 03:13:58.019169 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:57.996766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37983 (local address 127.0.0.1:35498)
0924 03:13:58.008949 (+ 12183us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:58.008974 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:13:58.009152 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:13:58.010505 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:13:58.010513 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:13:58.010552 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:13:58.011244 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:58.011261 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.013800 (+ 2539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.013805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:58.016058 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:58.016070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.018324 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.018333 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:58.018504 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:58.018535 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 03:13:58.018636 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 03:13:58.018766 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11990,"thread_start_us":1915,"threads_started":1}
I20260924 03:13:58.020258 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:57.998155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35498 (local address 127.0.97.3:37983)
0924 03:13:58.000877 (+ 2722us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:58.000885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:58.009331 (+ 8446us) server_negotiation.cc:407] Connection header received
0924 03:13:58.009472 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:58.009484 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:58.009889 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:58.010138 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:13:58.011983 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.013610 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.016256 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.017761 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.017904 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:58.018164 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:58.019734 (+ 1570us) server_negotiation.cc:299] Negotiation successful
0924 03:13:58.019924 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2543,"thread_start_us":2392,"threads_started":1}
I20260924 03:13:58.021641 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:58.001700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35233 (local address 127.0.0.1:36880)
0924 03:13:58.011775 (+ 10075us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:58.011794 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:13:58.011887 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:13:58.013293 (+ 1406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:13:58.013300 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:13:58.013338 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:13:58.013582 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:58.013594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.015868 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.015873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:58.017899 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:58.017914 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.020917 (+ 3003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.020923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:58.021091 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:58.021124 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 03:13:58.021194 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 03:13:58.021371 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9951,"threads_started":1}
I20260924 03:13:58.022760 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:58.002048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36880 (local address 127.0.97.2:35233)
0924 03:13:58.012436 (+ 10388us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:58.012445 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:58.012479 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:13:58.012598 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:58.012606 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:58.012846 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:58.013825 (+ 979us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:13:58.014065 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.015686 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.018100 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.020728 (+ 2628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.021920 (+ 1192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:58.022141 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:58.022221 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 03:13:58.022426 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10186,"thread_start_us":10927,"threads_started":1}
I20260924 03:13:58.022739 17703 tablet_service.cc:1467] Processing CreateTablet for tablet cfeff85d88614afb9f6dcfc47fb518d3 (DEFAULT_TABLE table=test-table [id=766ac766ba2246129076bb111589c218]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:13:58.024062 17971 tablet_service.cc:1467] Processing CreateTablet for tablet cfeff85d88614afb9f6dcfc47fb518d3 (DEFAULT_TABLE table=test-table [id=766ac766ba2246129076bb111589c218]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:13:58.025481 17971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfeff85d88614afb9f6dcfc47fb518d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:13:58.025892 17703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfeff85d88614afb9f6dcfc47fb518d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:13:58.036676 17837 tablet_service.cc:1467] Processing CreateTablet for tablet cfeff85d88614afb9f6dcfc47fb518d3 (DEFAULT_TABLE table=test-table [id=766ac766ba2246129076bb111589c218]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:13:58.038205 17837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfeff85d88614afb9f6dcfc47fb518d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:13:58.039405 18055 tablet_bootstrap.cc:492] T cfeff85d88614afb9f6dcfc47fb518d3 P e6fc21553ffa411497a89666805faf99: Bootstrap starting.
I20260924 03:13:58.042490 18055 tablet_bootstrap.cc:654] T cfeff85d88614afb9f6dcfc47fb518d3 P e6fc21553ffa411497a89666805faf99: Neither blocks nor log segments found. Creating new log.
I20260924 03:13:58.043545 18055 log.cc:824] T cfeff85d88614afb9f6dcfc47fb518d3 P e6fc21553ffa411497a89666805faf99: Log is configured to *not* fsync() on all Append() calls
I20260924 03:13:58.044443 18056 tablet_bootstrap.cc:492] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb: Bootstrap starting.
I20260924 03:13:58.048364 18056 tablet_bootstrap.cc:654] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb: Neither blocks nor log segments found. Creating new log.
I20260924 03:13:58.048612 18055 tablet_bootstrap.cc:492] T cfeff85d88614afb9f6dcfc47fb518d3 P e6fc21553ffa411497a89666805faf99: No bootstrap required, opened a new log
I20260924 03:13:58.048913 18055 ts_tablet_manager.cc:1397] T cfeff85d88614afb9f6dcfc47fb518d3 P e6fc21553ffa411497a89666805faf99: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 03:13:58.049706 18056 log.cc:824] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb: Log is configured to *not* fsync() on all Append() calls
I20260924 03:13:58.053086 18056 tablet_bootstrap.cc:492] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb: No bootstrap required, opened a new log
I20260924 03:13:58.053433 18059 tablet_bootstrap.cc:492] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1: Bootstrap starting.
I20260924 03:13:58.053481 18056 ts_tablet_manager.cc:1397] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260924 03:13:58.056716 18059 tablet_bootstrap.cc:654] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1: Neither blocks nor log segments found. Creating new log.
I20260924 03:13:58.058194 18059 log.cc:824] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1: Log is configured to *not* fsync() on all Append() calls
I20260924 03:13:58.060899 18055 raft_consensus.cc:348] T cfeff85d88614afb9f6dcfc47fb518d3 P e6fc21553ffa411497a89666805faf99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fc21553ffa411497a89666805faf99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "fda50413e68d4d5daaadc487da03e9bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37983 } } peers { permanent_uuid: "11e76ed6fa3d4812ba5369c27cbf55a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35233 } }
I20260924 03:13:58.061373 18055 raft_consensus.cc:374] T cfeff85d88614afb9f6dcfc47fb518d3 P e6fc21553ffa411497a89666805faf99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:13:58.061501 18055 raft_consensus.cc:729] T cfeff85d88614afb9f6dcfc47fb518d3 P e6fc21553ffa411497a89666805faf99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6fc21553ffa411497a89666805faf99, State: Initialized, Role: FOLLOWER
I20260924 03:13:58.062150 18055 consensus_queue.cc:260] T cfeff85d88614afb9f6dcfc47fb518d3 P e6fc21553ffa411497a89666805faf99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fc21553ffa411497a89666805faf99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "fda50413e68d4d5daaadc487da03e9bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37983 } } peers { permanent_uuid: "11e76ed6fa3d4812ba5369c27cbf55a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35233 } }
I20260924 03:13:58.064028 18055 ts_tablet_manager.cc:1428] T cfeff85d88614afb9f6dcfc47fb518d3 P e6fc21553ffa411497a89666805faf99: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260924 03:13:58.064265 18056 raft_consensus.cc:348] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fc21553ffa411497a89666805faf99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "fda50413e68d4d5daaadc487da03e9bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37983 } } peers { permanent_uuid: "11e76ed6fa3d4812ba5369c27cbf55a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35233 } }
I20260924 03:13:58.064764 18056 raft_consensus.cc:374] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:13:58.064889 18056 raft_consensus.cc:729] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda50413e68d4d5daaadc487da03e9bb, State: Initialized, Role: FOLLOWER
I20260924 03:13:58.065493 18056 consensus_queue.cc:260] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fc21553ffa411497a89666805faf99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "fda50413e68d4d5daaadc487da03e9bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37983 } } peers { permanent_uuid: "11e76ed6fa3d4812ba5369c27cbf55a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35233 } }
I20260924 03:13:58.066605 18036 heartbeater.cc:503] Master 127.0.97.62:39403 was elected leader, sending a full tablet report...
I20260924 03:13:58.067337 18056 ts_tablet_manager.cc:1428] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20260924 03:13:58.069849 18059 tablet_bootstrap.cc:492] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1: No bootstrap required, opened a new log
I20260924 03:13:58.070155 18059 ts_tablet_manager.cc:1397] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.010s
I20260924 03:13:58.077346 18059 raft_consensus.cc:348] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fc21553ffa411497a89666805faf99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "fda50413e68d4d5daaadc487da03e9bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37983 } } peers { permanent_uuid: "11e76ed6fa3d4812ba5369c27cbf55a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35233 } }
I20260924 03:13:58.077752 18059 raft_consensus.cc:374] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:13:58.077869 18059 raft_consensus.cc:729] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11e76ed6fa3d4812ba5369c27cbf55a1, State: Initialized, Role: FOLLOWER
I20260924 03:13:58.078419 18059 consensus_queue.cc:260] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fc21553ffa411497a89666805faf99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "fda50413e68d4d5daaadc487da03e9bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37983 } } peers { permanent_uuid: "11e76ed6fa3d4812ba5369c27cbf55a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35233 } }
W20260924 03:13:58.078625 17769 tablet.cc:2369] T cfeff85d88614afb9f6dcfc47fb518d3 P e6fc21553ffa411497a89666805faf99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:13:58.079552 17902 heartbeater.cc:503] Master 127.0.97.62:39403 was elected leader, sending a full tablet report...
I20260924 03:13:58.080178 18059 ts_tablet_manager.cc:1428] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
W20260924 03:13:58.120359 17903 tablet.cc:2369] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:13:58.155124 18062 raft_consensus.cc:482] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:13:58.155459 18062 raft_consensus.cc:504] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fc21553ffa411497a89666805faf99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "fda50413e68d4d5daaadc487da03e9bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37983 } } peers { permanent_uuid: "11e76ed6fa3d4812ba5369c27cbf55a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35233 } }
I20260924 03:13:58.157392 18062 leader_election.cc:288] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6fc21553ffa411497a89666805faf99 (127.0.97.1:37367), 11e76ed6fa3d4812ba5369c27cbf55a1 (127.0.97.2:35233)
W20260924 03:13:58.158849 18037 tablet.cc:2369] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:13:58.163959 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:58.157964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50895 (local address 127.0.97.2:35233)
0924 03:13:58.158185 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:58.158191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:58.158217 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:13:58.158306 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:58.158315 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:58.158445 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:58.158556 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:13:58.159024 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.160117 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.161918 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.163122 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.163258 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:58.163361 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:58.163631 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 03:13:58.163765 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 03:13:58.163998 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:58.157767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35233 (local address 127.0.97.3:50895)
0924 03:13:58.158003 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:58.158029 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 03:13:58.158171 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:13:58.158600 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:13:58.158605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:13:58.158637 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:13:58.158850 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:58.158863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.160293 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.160302 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:58.161755 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:58.161767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.163284 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.163288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:58.163468 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:58.163496 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 03:13:58.163570 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 03:13:58.163695 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":27}
I20260924 03:13:58.165099 17857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfeff85d88614afb9f6dcfc47fb518d3" candidate_uuid: "fda50413e68d4d5daaadc487da03e9bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e76ed6fa3d4812ba5369c27cbf55a1" is_pre_election: true
I20260924 03:13:58.165705 17857 raft_consensus.cc:2454] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fda50413e68d4d5daaadc487da03e9bb in term 0.
I20260924 03:13:58.165920 18064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:58.157788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.97.3:44485)
0924 03:13:58.159428 (+ 1640us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:58.159448 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:13:58.159569 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:13:58.159921 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:13:58.159925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:13:58.159950 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:13:58.160380 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:58.160397 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.161559 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.161566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:58.163540 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:58.163578 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.165222 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.165226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:58.165338 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:58.165365 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 03:13:58.165487 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 03:13:58.165605 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1486,"mutex_wait_us":296,"thread_start_us":388,"threads_started":1}
I20260924 03:13:58.165975 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:58.157954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44485 (local address 127.0.97.1:37367)
0924 03:13:58.158153 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:58.158159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:58.159543 (+ 1384us) server_negotiation.cc:407] Connection header received
0924 03:13:58.159726 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:58.159730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:58.159795 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:58.159872 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:13:58.160583 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.161396 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.163735 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.165030 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.165180 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:58.165303 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:58.165531 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 03:13:58.165656 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:13:58.166477 17924 leader_election.cc:302] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11e76ed6fa3d4812ba5369c27cbf55a1, fda50413e68d4d5daaadc487da03e9bb; no voters:
I20260924 03:13:58.166917 18062 raft_consensus.cc:2781] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:13:58.167083 18062 raft_consensus.cc:482] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:13:58.167198 18062 raft_consensus.cc:3037] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:13:58.167186 17723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfeff85d88614afb9f6dcfc47fb518d3" candidate_uuid: "fda50413e68d4d5daaadc487da03e9bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6fc21553ffa411497a89666805faf99" is_pre_election: true
I20260924 03:13:58.167748 17723 raft_consensus.cc:2454] T cfeff85d88614afb9f6dcfc47fb518d3 P e6fc21553ffa411497a89666805faf99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fda50413e68d4d5daaadc487da03e9bb in term 0.
I20260924 03:13:58.170789 18062 raft_consensus.cc:504] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fc21553ffa411497a89666805faf99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "fda50413e68d4d5daaadc487da03e9bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37983 } } peers { permanent_uuid: "11e76ed6fa3d4812ba5369c27cbf55a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35233 } }
I20260924 03:13:58.171552 18062 leader_election.cc:288] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb [CANDIDATE]: Term 1 election: Requested vote from peers e6fc21553ffa411497a89666805faf99 (127.0.97.1:37367), 11e76ed6fa3d4812ba5369c27cbf55a1 (127.0.97.2:35233)
I20260924 03:13:58.172091 17723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfeff85d88614afb9f6dcfc47fb518d3" candidate_uuid: "fda50413e68d4d5daaadc487da03e9bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6fc21553ffa411497a89666805faf99"
I20260924 03:13:58.172215 17857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfeff85d88614afb9f6dcfc47fb518d3" candidate_uuid: "fda50413e68d4d5daaadc487da03e9bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e76ed6fa3d4812ba5369c27cbf55a1"
I20260924 03:13:58.172327 17723 raft_consensus.cc:3037] T cfeff85d88614afb9f6dcfc47fb518d3 P e6fc21553ffa411497a89666805faf99 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:13:58.172483 17857 raft_consensus.cc:3037] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:13:58.174656 17723 raft_consensus.cc:2454] T cfeff85d88614afb9f6dcfc47fb518d3 P e6fc21553ffa411497a89666805faf99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fda50413e68d4d5daaadc487da03e9bb in term 1.
I20260924 03:13:58.174702 17857 raft_consensus.cc:2454] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fda50413e68d4d5daaadc487da03e9bb in term 1.
I20260924 03:13:58.175122 17927 leader_election.cc:302] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6fc21553ffa411497a89666805faf99, fda50413e68d4d5daaadc487da03e9bb; no voters:
I20260924 03:13:58.175465 18062 raft_consensus.cc:2781] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:13:58.175966 18062 raft_consensus.cc:686] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb [term 1 LEADER]: Becoming Leader. State: Replica: fda50413e68d4d5daaadc487da03e9bb, State: Running, Role: LEADER
I20260924 03:13:58.176822 18062 consensus_queue.cc:237] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fc21553ffa411497a89666805faf99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "fda50413e68d4d5daaadc487da03e9bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37983 } } peers { permanent_uuid: "11e76ed6fa3d4812ba5369c27cbf55a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35233 } }
I20260924 03:13:58.181329 17578 catalog_manager.cc:5495] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb reported cstate change: term changed from 0 to 1, leader changed from <none> to fda50413e68d4d5daaadc487da03e9bb (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fda50413e68d4d5daaadc487da03e9bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fc21553ffa411497a89666805faf99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fda50413e68d4d5daaadc487da03e9bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11e76ed6fa3d4812ba5369c27cbf55a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20260924 03:13:58.200451 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:58.193066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35514 (local address 127.0.97.3:37983)
0924 03:13:58.193361 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:58.193367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:58.193461 (+ 94us) server_negotiation.cc:407] Connection header received
0924 03:13:58.193752 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:58.193758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:58.193915 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:58.194116 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:13:58.194602 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.195927 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.198149 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.198626 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.198764 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:58.199310 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:13:58.199834 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:13:58.200011 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:58.200091 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 03:13:58.200176 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 03:13:58.208649 17857 raft_consensus.cc:1261] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1 [term 1 FOLLOWER]: Refusing update from remote peer fda50413e68d4d5daaadc487da03e9bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:13:58.208922 17723 raft_consensus.cc:1261] T cfeff85d88614afb9f6dcfc47fb518d3 P e6fc21553ffa411497a89666805faf99 [term 1 FOLLOWER]: Refusing update from remote peer fda50413e68d4d5daaadc487da03e9bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:13:58.209681 18065 consensus_queue.cc:1035] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "11e76ed6fa3d4812ba5369c27cbf55a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:13:58.210285 18062 consensus_queue.cc:1035] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6fc21553ffa411497a89666805faf99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:13:58.236330 17578 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:13:58.236706 17578 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:13:58.240677 17578 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 861bf5cc313440ab84d38812c88e988f: Sending DeleteTablet for 3 replicas of tablet cfeff85d88614afb9f6dcfc47fb518d3
I20260924 03:13:58.241964 17971 tablet_service.cc:1514] Processing DeleteTablet for tablet cfeff85d88614afb9f6dcfc47fb518d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:13:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35498
I20260924 03:13:58.242024 17837 tablet_service.cc:1514] Processing DeleteTablet for tablet cfeff85d88614afb9f6dcfc47fb518d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:13:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36880
I20260924 03:13:58.242085 17703 tablet_service.cc:1514] Processing DeleteTablet for tablet cfeff85d88614afb9f6dcfc47fb518d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:13:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50318
I20260924 03:13:58.246200 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 17640
I20260924 03:13:58.253026 18080 tablet_replica.cc:330] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1: stopping tablet replica
I20260924 03:13:58.253309 18078 tablet_replica.cc:330] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb: stopping tablet replica
I20260924 03:13:58.254017 18078 raft_consensus.cc:2229] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:13:58.254309 18080 raft_consensus.cc:2229] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:13:58.255051 18078 raft_consensus.cc:2258] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:13:58.255932 18080 raft_consensus.cc:2258] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:13:58.257980 18078 ts_tablet_manager.cc:1905] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:13:58.259876 18080 ts_tablet_manager.cc:1905] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:13:58.267311 18078 ts_tablet_manager.cc:1918] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:13:58.267566 18078 log.cc:1196] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/cfeff85d88614afb9f6dcfc47fb518d3
I20260924 03:13:58.268321 18078 ts_tablet_manager.cc:1939] T cfeff85d88614afb9f6dcfc47fb518d3 P fda50413e68d4d5daaadc487da03e9bb: Deleting consensus metadata
I20260924 03:13:58.270038 18080 ts_tablet_manager.cc:1918] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:13:58.270318 18080 log.cc:1196] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/cfeff85d88614afb9f6dcfc47fb518d3
I20260924 03:13:58.270592 17564 catalog_manager.cc:4841] TS fda50413e68d4d5daaadc487da03e9bb (127.0.97.3:37983): tablet cfeff85d88614afb9f6dcfc47fb518d3 (table test-table [id=766ac766ba2246129076bb111589c218]) successfully deleted
I20260924 03:13:58.271090 18079 tablet_replica.cc:330] T cfeff85d88614afb9f6dcfc47fb518d3 P e6fc21553ffa411497a89666805faf99: stopping tablet replica
Sep 24 03:13:58 dist-test-slave-5l98 krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:13:58.271551 18079 raft_consensus.cc:2229] T cfeff85d88614afb9f6dcfc47fb518d3 P e6fc21553ffa411497a89666805faf99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:13:58.271903 18080 ts_tablet_manager.cc:1939] T cfeff85d88614afb9f6dcfc47fb518d3 P 11e76ed6fa3d4812ba5369c27cbf55a1: Deleting consensus metadata
I20260924 03:13:58.272169 18079 raft_consensus.cc:2258] T cfeff85d88614afb9f6dcfc47fb518d3 P e6fc21553ffa411497a89666805faf99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:13:58.274112 17563 catalog_manager.cc:4841] TS 11e76ed6fa3d4812ba5369c27cbf55a1 (127.0.97.2:35233): tablet cfeff85d88614afb9f6dcfc47fb518d3 (table test-table [id=766ac766ba2246129076bb111589c218]) successfully deleted
I20260924 03:13:58.274693 18079 ts_tablet_manager.cc:1905] T cfeff85d88614afb9f6dcfc47fb518d3 P e6fc21553ffa411497a89666805faf99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:13:58.280331 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:58.247962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50330 (local address 127.0.97.1:37367)
0924 03:13:58.248147 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:58.248156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:58.259737 (+ 11581us) server_negotiation.cc:407] Connection header received
0924 03:13:58.260317 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:58.260323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:58.260447 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:58.260549 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:13:58.262473 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.263657 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.265803 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.266494 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.266600 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:58.273036 (+ 6436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:13:58.273053 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:13:58.273076 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:13:58.273154 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:13:58.277155 (+ 4001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:58.277964 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:58.277971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:58.277974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:58.278125 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:58.278549 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:58.278558 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:58.278560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:58.278960 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:13:58.279318 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:58.279661 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 03:13:58.279936 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:13:58.281118 17763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:13:58.377270 18079 ts_tablet_manager.cc:1918] T cfeff85d88614afb9f6dcfc47fb518d3 P e6fc21553ffa411497a89666805faf99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:13:58.377812 18079 log.cc:1196] T cfeff85d88614afb9f6dcfc47fb518d3 P e6fc21553ffa411497a89666805faf99: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/cfeff85d88614afb9f6dcfc47fb518d3
I20260924 03:13:58.377995 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 17640
I20260924 03:13:58.397773 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 17774
W20260924 03:13:58.397863 17566 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:37367 (error 108)
I20260924 03:13:58.400302 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:58.399766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.0.1:50338)
0924 03:13:58.399970 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:58.400099 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260924 03:13:58.400664 17566 catalog_manager.cc:4584] TS e6fc21553ffa411497a89666805faf99 (127.0.97.1:37367): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cfeff85d88614afb9f6dcfc47fb518d3: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37367: connect: Connection refused (error 111)
Sep 24 03:13:58 dist-test-slave-5l98 krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 03:13:58.416370 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:58.399042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.0.97.2:35233)
0924 03:13:58.399237 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:58.399243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:58.399265 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:13:58.399328 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:58.399332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:58.399411 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:58.399496 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:13:58.401271 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.402115 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.404081 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.404482 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.404734 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:58.410141 (+ 5407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:13:58.410155 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:13:58.410171 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:13:58.410243 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:13:58.413447 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:58.414178 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:58.414185 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:58.414187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:58.414328 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:58.414711 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:58.414716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:58.414719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:58.415130 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:13:58.415417 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:58.415747 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 03:13:58.415977 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 03:13:58.417135 17897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:13:58.454424 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:58.453867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.0.1:50340)
0924 03:13:58.454125 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:58.454238 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 03:13:58.455690 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:58.455236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.0.1:50344)
0924 03:13:58.455443 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:58.455527 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 03:13:58.509632 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 17774
I20260924 03:13:58.516705 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:58.516059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.0.1:50358)
0924 03:13:58.516342 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:58.516463 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260924 03:13:58.518078 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:58.517576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.0.1:50360)
0924 03:13:58.517776 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:58.517879 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 03:13:58.527532 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 17908
Sep 24 03:13:58 dist-test-slave-5l98 krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 03:13:58.546742 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:58.528474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35526 (local address 127.0.97.3:37983)
0924 03:13:58.528691 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:58.528698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:58.528742 (+ 44us) server_negotiation.cc:407] Connection header received
0924 03:13:58.528812 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:58.528818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:58.528912 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:58.529028 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:13:58.530437 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.531680 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.533343 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:58.534024 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:58.534180 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:58.538707 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:13:58.538721 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:13:58.538742 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:13:58.538801 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:13:58.543143 (+ 4342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:58.544042 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:58.544053 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:58.544057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:58.544237 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:13:58.544863 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:13:58.544868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:13:58.544871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:13:58.545396 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:13:58.545730 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:58.546077 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 03:13:58.546344 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 03:13:58.547709 18031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:13:58.598546 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:58.598089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.0.1:50362)
0924 03:13:58.598293 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:58.598389 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 03:13:58.599609 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:58.599208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.0.1:50374)
0924 03:13:58.599439 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:58.599506 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260924 03:13:58.674585 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 17908
I20260924 03:13:58.694790 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 17547
I20260924 03:13:58.696126 17609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:13:58.809763 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:58.809230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.0.1:50382)
0924 03:13:58.809427 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:58.809552 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 03:13:58.809737 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 17547
2026-09-24T03:13:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4548 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T03:13:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:13:58Z Disabled control of system clock
I20260924 03:13:58.887979 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42535
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:13:59.224383 18090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:13:59.224800 18090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:13:59.224896 18090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:13:59.232388 18090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:13:59.232548 18090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:13:59.232610 18090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:13:59.232673 18090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:13:59.232718 18090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:13:59.242218 18090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:13:59.243500 18090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:13:59.245304 18090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:13:59.257411 18095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:59.257316 18098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:59.257318 18096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:13:59.258862 18090 server_base.cc:1034] running on GCE node
I20260924 03:13:59.259670 18090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:13:59.261209 18090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:13:59.262456 18090 hybrid_clock.cc:648] HybridClock initialized: now 1790219639262408 us; error 63 us; skew 500 ppm
I20260924 03:13:59.262887 18090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:13:59.269821 18090 webserver.cc:460] Webserver started at http://127.0.97.62:35489/ using document root <none> and password file <none>
I20260924 03:13:59.270565 18090 fs_manager.cc:359] Metadata directory not provided
I20260924 03:13:59.270706 18090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:13:59.271042 18090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:13:59.274156 18090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "795d98a846d440f790e5125d89004655"
format_stamp: "Formatted at 2026-09-24 03:13:59 on dist-test-slave-5l98"
server_key: "42e81c527bd67120ab78ccc8e4185f41"
server_key_iv: "42e19bf0c4e3a46fac911f107e25971c"
server_key_version: "encryptionkey@0"
I20260924 03:13:59.275007 18090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "795d98a846d440f790e5125d89004655"
format_stamp: "Formatted at 2026-09-24 03:13:59 on dist-test-slave-5l98"
server_key: "42e81c527bd67120ab78ccc8e4185f41"
server_key_iv: "42e19bf0c4e3a46fac911f107e25971c"
server_key_version: "encryptionkey@0"
I20260924 03:13:59.280444 18090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:13:59.284121 18104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:59.285646 18090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:13:59.285848 18090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "795d98a846d440f790e5125d89004655"
format_stamp: "Formatted at 2026-09-24 03:13:59 on dist-test-slave-5l98"
server_key: "42e81c527bd67120ab78ccc8e4185f41"
server_key_iv: "42e19bf0c4e3a46fac911f107e25971c"
server_key_version: "encryptionkey@0"
I20260924 03:13:59.286077 18090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:13:59.324049 18090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:13:59.324965 18090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:13:59.325260 18090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:13:59.344795 18090 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42535
I20260924 03:13:59.344843 18155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42535 every 8 connection(s)
I20260924 03:13:59.346282 18090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:13:59.350626 18156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:13:59.353121 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 18090
I20260924 03:13:59.353318 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:13:59.353746 388 external_mini_cluster.cc:1428] Setting key 68c2367851fc5b0a8152e6e2ce32756b
I20260924 03:13:59.364640 18156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655: Bootstrap starting.
I20260924 03:13:59.368914 18156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655: Neither blocks nor log segments found. Creating new log.
I20260924 03:13:59.370118 18156 log.cc:824] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655: Log is configured to *not* fsync() on all Append() calls
I20260924 03:13:59.370611 18159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:59.355848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.0.97.62:42535)
0924 03:13:59.356987 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:59.357010 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:59.357065 (+ 55us) server_negotiation.cc:407] Connection header received
0924 03:13:59.359443 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:59.359471 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:59.360049 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:59.360551 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:13:59.361138 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:59.363298 (+ 2160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:59.364988 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:59.366013 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:59.366210 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:59.367878 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:13:59.367894 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:13:59.367918 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:13:59.368659 (+ 741us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:13:59.368773 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:13:59.368803 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:13:59.368928 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:59.369208 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 03:13:59.369556 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":361,"threads_started":1}
I20260924 03:13:59.373761 18156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655: No bootstrap required, opened a new log
I20260924 03:13:59.381023 18156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795d98a846d440f790e5125d89004655" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42535 } }
I20260924 03:13:59.381413 18156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:13:59.381523 18156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795d98a846d440f790e5125d89004655, State: Initialized, Role: FOLLOWER
I20260924 03:13:59.382059 18156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795d98a846d440f790e5125d89004655" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42535 } }
I20260924 03:13:59.382303 18156 raft_consensus.cc:388] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:13:59.382449 18156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:13:59.382619 18156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:13:59.385151 18156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795d98a846d440f790e5125d89004655" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42535 } }
I20260924 03:13:59.385653 18156 leader_election.cc:302] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 795d98a846d440f790e5125d89004655; no voters:
I20260924 03:13:59.386233 18156 leader_election.cc:288] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:13:59.386433 18161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:13:59.387389 18161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655 [term 1 LEADER]: Becoming Leader. State: Replica: 795d98a846d440f790e5125d89004655, State: Running, Role: LEADER
I20260924 03:13:59.387959 18161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795d98a846d440f790e5125d89004655" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42535 } }
I20260924 03:13:59.388476 18156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:13:59.391091 18162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "795d98a846d440f790e5125d89004655" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795d98a846d440f790e5125d89004655" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42535 } } }
I20260924 03:13:59.391597 18162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655 [sys.catalog]: This master's current role is: LEADER
I20260924 03:13:59.392220 18163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 795d98a846d440f790e5125d89004655. Latest consensus state: current_term: 1 leader_uuid: "795d98a846d440f790e5125d89004655" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795d98a846d440f790e5125d89004655" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42535 } } }
I20260924 03:13:59.392616 18163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655 [sys.catalog]: This master's current role is: LEADER
I20260924 03:13:59.393970 18167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:13:59.399712 18167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:13:59.409356 18167 catalog_manager.cc:1347] Generated new cluster ID: 8da0a1d663f04010a6d35c2808df7254
I20260924 03:13:59.409490 18167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:13:59.439208 18167 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:13:59.441179 18167 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:13:59.451398 18167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655: Generated new TSK 0
I20260924 03:13:59.452605 18167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:13:59.471879 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42535
--builtin_ntp_servers=127.0.97.20:46793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:13:59.817926 18180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:13:59.818214 18180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:13:59.818370 18180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:13:59.825843 18180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:13:59.825951 18180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:13:59.826135 18180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:13:59.835462 18180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:13:59.836903 18180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:13:59.838670 18180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:13:59.853904 18185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:59.853869 18186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:13:59.853940 18188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:13:59.854377 18180 server_base.cc:1034] running on GCE node
I20260924 03:13:59.855329 18180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:13:59.856402 18180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:13:59.857725 18180 hybrid_clock.cc:648] HybridClock initialized: now 1790219639857626 us; error 86 us; skew 500 ppm
I20260924 03:13:59.858027 18180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:13:59.860857 18180 webserver.cc:460] Webserver started at http://127.0.97.1:42147/ using document root <none> and password file <none>
I20260924 03:13:59.861532 18180 fs_manager.cc:359] Metadata directory not provided
I20260924 03:13:59.861624 18180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:13:59.861902 18180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:13:59.864892 18180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "d01839204681437abd135beb7616966e"
format_stamp: "Formatted at 2026-09-24 03:13:59 on dist-test-slave-5l98"
server_key: "112016c4fe78d1edce14700645380c2b"
server_key_iv: "2e9b3a2bbaa0d6e2fbe97d2d5513c785"
server_key_version: "encryptionkey@0"
I20260924 03:13:59.865772 18180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d01839204681437abd135beb7616966e"
format_stamp: "Formatted at 2026-09-24 03:13:59 on dist-test-slave-5l98"
server_key: "112016c4fe78d1edce14700645380c2b"
server_key_iv: "2e9b3a2bbaa0d6e2fbe97d2d5513c785"
server_key_version: "encryptionkey@0"
I20260924 03:13:59.871414 18180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 03:13:59.875085 18194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:59.876719 18180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:13:59.876911 18180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "d01839204681437abd135beb7616966e"
format_stamp: "Formatted at 2026-09-24 03:13:59 on dist-test-slave-5l98"
server_key: "112016c4fe78d1edce14700645380c2b"
server_key_iv: "2e9b3a2bbaa0d6e2fbe97d2d5513c785"
server_key_version: "encryptionkey@0"
I20260924 03:13:59.877148 18180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:13:59.899621 18180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:13:59.900429 18180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:13:59.900763 18180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:13:59.901726 18180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:13:59.903278 18180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:13:59.903434 18180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:59.903558 18180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:13:59.903685 18180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:13:59.940328 18180 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41475
I20260924 03:13:59.940421 18306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41475 every 8 connection(s)
I20260924 03:13:59.941685 18180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:13:59.947711 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 18180
I20260924 03:13:59.947899 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:13:59.948421 388 external_mini_cluster.cc:1428] Setting key 3b0a3ceed452fbc7e43e5a2c6f122601
I20260924 03:13:59.951633 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42535
--builtin_ntp_servers=127.0.97.20:46793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:13:59.955286 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:59.944406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54209 (local address 127.0.97.62:42535)
0924 03:13:59.945000 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0924 03:13:59.945004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:13:59.946176 (+ 1172us) server_negotiation.cc:407] Connection header received
0924 03:13:59.948136 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:13:59.948142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:13:59.948242 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:13:59.948365 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:13:59.949585 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:59.950482 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:59.952368 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:13:59.952850 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:13:59.952987 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:59.954393 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:13:59.954398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:13:59.954402 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:13:59.954422 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:13:59.954461 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:13:59.954472 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:13:59.954567 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 03:13:59.954807 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 03:13:59.954907 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":375,"threads_started":1}
I20260924 03:13:59.956281 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:13:59.945079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42535 (local address 127.0.97.1:54209)
0924 03:13:59.946015 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0924 03:13:59.946051 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:13:59.947775 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:13:59.948639 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:13:59.948653 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:13:59.948743 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:13:59.949420 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:59.949438 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:59.950687 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:59.950691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:59.952224 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:13:59.952236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:13:59.953071 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:13:59.953078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:13:59.953332 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:13:59.954046 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:13:59.954049 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:13:59.954089 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:13:59.954173 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:13:59.954580 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:13:59.954593 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:13:59.954608 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 03:13:59.954799 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 03:13:59.955210 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":278,"threads_started":1}
I20260924 03:13:59.958021 18307 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42535
I20260924 03:13:59.958381 18307 heartbeater.cc:464] Registering TS with master...
I20260924 03:13:59.959151 18307 heartbeater.cc:511] Master 127.0.97.62:42535 requested a full tablet report, sending...
I20260924 03:13:59.961329 18121 ts_manager.cc:194] Registered new tserver with Master: d01839204681437abd135beb7616966e (127.0.97.1:41475)
I20260924 03:13:59.962752 18121 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:54209
W20260924 03:14:00.310319 18311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:00.310693 18311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:00.310878 18311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:00.319211 18311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:00.319372 18311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:00.319557 18311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:14:00.329387 18311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:00.330880 18311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:00.332744 18311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:00.347049 18316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:00.347725 18319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:00.347932 18317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:00.347891 18311 server_base.cc:1034] running on GCE node
I20260924 03:14:00.348702 18311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:00.349782 18311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:00.351045 18311 hybrid_clock.cc:648] HybridClock initialized: now 1790219640350990 us; error 48 us; skew 500 ppm
I20260924 03:14:00.351389 18311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:14:00.353916 18311 webserver.cc:460] Webserver started at http://127.0.97.2:42713/ using document root <none> and password file <none>
I20260924 03:14:00.354643 18311 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:00.354781 18311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:00.355104 18311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:00.358094 18311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "7e43573e4ba341e99b6f1891741e430e"
format_stamp: "Formatted at 2026-09-24 03:14:00 on dist-test-slave-5l98"
server_key: "164a124e8ca581c7d49f160c51d5dc70"
server_key_iv: "0a6c67b9169f86bbed941a5158e6b6e3"
server_key_version: "encryptionkey@0"
I20260924 03:14:00.358929 18311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7e43573e4ba341e99b6f1891741e430e"
format_stamp: "Formatted at 2026-09-24 03:14:00 on dist-test-slave-5l98"
server_key: "164a124e8ca581c7d49f160c51d5dc70"
server_key_iv: "0a6c67b9169f86bbed941a5158e6b6e3"
server_key_version: "encryptionkey@0"
I20260924 03:14:00.364456 18311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 03:14:00.368319 18325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:00.369988 18311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:14:00.370190 18311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "7e43573e4ba341e99b6f1891741e430e"
format_stamp: "Formatted at 2026-09-24 03:14:00 on dist-test-slave-5l98"
server_key: "164a124e8ca581c7d49f160c51d5dc70"
server_key_iv: "0a6c67b9169f86bbed941a5158e6b6e3"
server_key_version: "encryptionkey@0"
I20260924 03:14:00.370436 18311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:00.395644 18311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:00.396543 18311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:00.396838 18311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:00.397868 18311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:14:00.399437 18311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:14:00.399567 18311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:00.399708 18311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:14:00.399798 18311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:00.435549 18311 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33599
I20260924 03:14:00.435642 18437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33599 every 8 connection(s)
I20260924 03:14:00.436965 18311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:14:00.438334 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 18311
I20260924 03:14:00.438517 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:14:00.438993 388 external_mini_cluster.cc:1428] Setting key 3c603864a68fabedfeb53c267bfff65a
I20260924 03:14:00.442418 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42535
--builtin_ntp_servers=127.0.97.20:46793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:14:00.452395 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:00.439822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35433 (local address 127.0.97.62:42535)
0924 03:14:00.440001 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:00.440005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:00.441814 (+ 1809us) server_negotiation.cc:407] Connection header received
0924 03:14:00.443820 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:00.443829 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:00.443925 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:00.444045 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:00.445378 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.446498 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.448762 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.449519 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.449698 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:00.451551 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:00.451560 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:00.451568 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:00.451604 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:00.451651 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:00.451666 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:00.451768 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:00.452037 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 03:14:00.452124 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 03:14:00.454041 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:00.440449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42535 (local address 127.0.97.2:35433)
0924 03:14:00.441653 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:00.441690 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:00.443526 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:00.444276 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:00.444284 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:00.444394 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:00.445191 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:00.445209 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.446885 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.446894 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:00.448569 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:00.448594 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.449791 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.449799 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:00.450016 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:00.451024 (+ 1008us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:14:00.451029 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:00.451099 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:14:00.451269 (+ 170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:00.451783 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:00.451797 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:00.451808 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 03:14:00.451995 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 03:14:00.452451 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":344,"threads_started":1}
I20260924 03:14:00.455881 18438 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42535
I20260924 03:14:00.456229 18438 heartbeater.cc:464] Registering TS with master...
I20260924 03:14:00.457064 18438 heartbeater.cc:511] Master 127.0.97.62:42535 requested a full tablet report, sending...
I20260924 03:14:00.458775 18121 ts_manager.cc:194] Registered new tserver with Master: 7e43573e4ba341e99b6f1891741e430e (127.0.97.2:33599)
I20260924 03:14:00.460078 18121 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:35433
W20260924 03:14:00.772365 18441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:00.772708 18441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:00.772799 18441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:00.780514 18441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:00.780622 18441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:00.780732 18441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:14:00.790277 18441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:00.791517 18441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:00.793304 18441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:00.807868 18446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:00.807870 18449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:00.807893 18447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:00.808961 18441 server_base.cc:1034] running on GCE node
I20260924 03:14:00.809581 18441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:00.810541 18441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:00.811751 18441 hybrid_clock.cc:648] HybridClock initialized: now 1790219640811704 us; error 49 us; skew 500 ppm
I20260924 03:14:00.812093 18441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:14:00.814776 18441 webserver.cc:460] Webserver started at http://127.0.97.3:45667/ using document root <none> and password file <none>
I20260924 03:14:00.815454 18441 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:00.815541 18441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:00.815810 18441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:00.818755 18441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "41bceec5148848e8b6b576303ce5effb"
format_stamp: "Formatted at 2026-09-24 03:14:00 on dist-test-slave-5l98"
server_key: "1c56dd318ad23ba991126017e7139870"
server_key_iv: "796120caf5511174a37584a60fa5caaf"
server_key_version: "encryptionkey@0"
I20260924 03:14:00.819522 18441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "41bceec5148848e8b6b576303ce5effb"
format_stamp: "Formatted at 2026-09-24 03:14:00 on dist-test-slave-5l98"
server_key: "1c56dd318ad23ba991126017e7139870"
server_key_iv: "796120caf5511174a37584a60fa5caaf"
server_key_version: "encryptionkey@0"
I20260924 03:14:00.824774 18441 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 03:14:00.828295 18455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:00.829798 18441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:14:00.829988 18441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "41bceec5148848e8b6b576303ce5effb"
format_stamp: "Formatted at 2026-09-24 03:14:00 on dist-test-slave-5l98"
server_key: "1c56dd318ad23ba991126017e7139870"
server_key_iv: "796120caf5511174a37584a60fa5caaf"
server_key_version: "encryptionkey@0"
I20260924 03:14:00.830179 18441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:00.844988 18441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:00.845821 18441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:00.846065 18441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:00.846987 18441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:14:00.848436 18441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:14:00.848584 18441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:00.848675 18441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:14:00.848730 18441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:00.885710 18441 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46255
I20260924 03:14:00.885864 18567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46255 every 8 connection(s)
I20260924 03:14:00.887053 18441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:14:00.891265 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 18441
I20260924 03:14:00.891454 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:14:00.892093 388 external_mini_cluster.cc:1428] Setting key 367cf71ba0f81183bb384a3dcd39b25a
I20260924 03:14:00.901574 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:00.889926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54799 (local address 127.0.97.62:42535)
0924 03:14:00.890130 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:00.890134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:00.891736 (+ 1602us) server_negotiation.cc:407] Connection header received
0924 03:14:00.893847 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:00.893852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:00.893962 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:00.894077 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:00.895772 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.896611 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.898497 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.899028 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.899143 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:00.900758 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:00.900764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:00.900767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:00.900789 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:00.900823 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:00.900836 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:00.900920 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:00.901222 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 03:14:00.901309 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 03:14:00.902552 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:00.890474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42535 (local address 127.0.97.3:54799)
0924 03:14:00.891587 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:00.891628 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:00.893530 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:00.894599 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:00.894611 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:00.894736 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:00.895595 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:00.895616 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.896755 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.896760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:00.898252 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:00.898266 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.899256 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.899261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:00.899520 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:00.900220 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:14:00.900223 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:00.900284 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:14:00.900400 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:00.900968 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:00.900977 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:00.900983 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 03:14:00.901150 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 03:14:00.901459 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":344,"threads_started":1}
I20260924 03:14:00.904296 18568 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42535
I20260924 03:14:00.904629 18568 heartbeater.cc:464] Registering TS with master...
I20260924 03:14:00.905409 18568 heartbeater.cc:511] Master 127.0.97.62:42535 requested a full tablet report, sending...
I20260924 03:14:00.907198 18121 ts_manager.cc:194] Registered new tserver with Master: 41bceec5148848e8b6b576303ce5effb (127.0.97.3:46255)
I20260924 03:14:00.908349 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:14:00.908401 18121 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:54799
I20260924 03:14:00.923225 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:00.916817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34138 (local address 127.0.97.62:42535)
0924 03:14:00.917012 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:00.917017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:00.917445 (+ 428us) server_negotiation.cc:407] Connection header received
0924 03:14:00.917575 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:00.917580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:00.917645 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:00.917838 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:00.918405 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.919225 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.920745 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.921196 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.921314 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:00.922486 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:00.922491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:00.922495 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:00.922513 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:00.922547 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:00.922559 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:00.922635 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:00.922899 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 03:14:00.922974 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:14:00.927592 18121 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:14:00.934759 18121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:14:00.965809 18307 heartbeater.cc:503] Master 127.0.97.62:42535 was elected leader, sending a full tablet report...
I20260924 03:14:00.975195 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:00.948629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52228 (local address 127.0.97.3:46255)
0924 03:14:00.963851 (+ 15222us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:00.963864 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:00.965608 (+ 1744us) server_negotiation.cc:407] Connection header received
0924 03:14:00.965814 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:00.965826 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:00.966114 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:00.967138 (+ 1024us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:00.967465 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.969401 (+ 1936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.972611 (+ 3210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.973838 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.974023 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:00.974332 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:00.974561 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 03:14:00.974820 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15026,"thread_start_us":14866,"threads_started":1}
I20260924 03:14:00.975313 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:00.948350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46255 (local address 127.0.0.1:52228)
0924 03:14:00.965487 (+ 17137us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:00.965509 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:00.965631 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:00.966405 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:00.966412 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:00.966442 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:00.966905 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:00.966922 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.969557 (+ 2635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.969565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:00.972328 (+ 2763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:00.972340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.974054 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.974060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:00.974271 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:00.974338 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 03:14:00.974513 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 03:14:00.974721 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16980,"mutex_wait_us":200,"thread_start_us":15616,"threads_started":1}
I20260924 03:14:00.975971 18581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:00.948059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33599 (local address 127.0.0.1:38526)
0924 03:14:00.964511 (+ 16452us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:00.964567 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:00.964831 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:00.966733 (+ 1902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:00.966742 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:00.966789 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:00.967403 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:00.967420 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.969858 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.969862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:00.972710 (+ 2848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:00.972725 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.974894 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.974900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:00.975120 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:00.975381 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 03:14:00.975476 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 03:14:00.975607 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16058,"mutex_wait_us":367,"thread_start_us":18015,"threads_started":1}
I20260924 03:14:00.976714 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:00.948536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38526 (local address 127.0.97.2:33599)
0924 03:14:00.966121 (+ 17585us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:00.966129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:00.966160 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:14:00.966276 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:00.966285 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:00.966491 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:00.966708 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:00.967694 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.969721 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.972930 (+ 3209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.974392 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.974515 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:00.974757 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:00.976209 (+ 1452us) server_negotiation.cc:299] Negotiation successful
0924 03:14:00.976379 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17425,"thread_start_us":18375,"threads_started":1}
I20260924 03:14:00.979650 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:00.967280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41475 (local address 127.0.0.1:48380)
0924 03:14:00.969508 (+ 2228us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:00.969527 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:00.971044 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:00.972307 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:00.972313 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:00.972340 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:00.972780 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:00.972789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.974723 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.974728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:00.977170 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:00.977183 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.978841 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.978848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:00.979045 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:00.979080 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 03:14:00.979168 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 03:14:00.979319 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2087,"thread_start_us":1929,"threads_started":1}
I20260924 03:14:00.980329 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:00.965377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48380 (local address 127.0.97.1:41475)
0924 03:14:00.969022 (+ 3645us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:00.969030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:00.970458 (+ 1428us) server_negotiation.cc:407] Connection header received
0924 03:14:00.971824 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:00.971833 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:00.972083 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:00.972283 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:00.972979 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.974544 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.977349 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:00.978623 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:00.978775 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:00.979042 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:00.979565 (+ 523us) server_negotiation.cc:299] Negotiation successful
0924 03:14:00.979750 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3463,"thread_start_us":3335,"threads_started":1}
I20260924 03:14:00.980903 18373 tablet_service.cc:1467] Processing CreateTablet for tablet 639f016d818d4ac2a6e45d0291d60154 (DEFAULT_TABLE table=test-table [id=de9e689008714fd8a449f9dfa11e6cc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:14:00.981081 18503 tablet_service.cc:1467] Processing CreateTablet for tablet 639f016d818d4ac2a6e45d0291d60154 (DEFAULT_TABLE table=test-table [id=de9e689008714fd8a449f9dfa11e6cc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:14:00.982580 18503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 639f016d818d4ac2a6e45d0291d60154. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:00.983825 18242 tablet_service.cc:1467] Processing CreateTablet for tablet 639f016d818d4ac2a6e45d0291d60154 (DEFAULT_TABLE table=test-table [id=de9e689008714fd8a449f9dfa11e6cc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:14:00.985102 18242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 639f016d818d4ac2a6e45d0291d60154. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:00.985116 18373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 639f016d818d4ac2a6e45d0291d60154. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:00.996018 18586 tablet_bootstrap.cc:492] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb: Bootstrap starting.
I20260924 03:14:00.999508 18586 tablet_bootstrap.cc:654] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:01.001134 18586 log.cc:824] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:01.004361 18589 tablet_bootstrap.cc:492] T 639f016d818d4ac2a6e45d0291d60154 P 7e43573e4ba341e99b6f1891741e430e: Bootstrap starting.
I20260924 03:14:01.006714 18587 tablet_bootstrap.cc:492] T 639f016d818d4ac2a6e45d0291d60154 P d01839204681437abd135beb7616966e: Bootstrap starting.
I20260924 03:14:01.008466 18586 tablet_bootstrap.cc:492] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb: No bootstrap required, opened a new log
I20260924 03:14:01.008677 18589 tablet_bootstrap.cc:654] T 639f016d818d4ac2a6e45d0291d60154 P 7e43573e4ba341e99b6f1891741e430e: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:01.009050 18586 ts_tablet_manager.cc:1397] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.002s
I20260924 03:14:01.010048 18589 log.cc:824] T 639f016d818d4ac2a6e45d0291d60154 P 7e43573e4ba341e99b6f1891741e430e: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:01.010774 18587 tablet_bootstrap.cc:654] T 639f016d818d4ac2a6e45d0291d60154 P d01839204681437abd135beb7616966e: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:01.011950 18587 log.cc:824] T 639f016d818d4ac2a6e45d0291d60154 P d01839204681437abd135beb7616966e: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:01.013448 18589 tablet_bootstrap.cc:492] T 639f016d818d4ac2a6e45d0291d60154 P 7e43573e4ba341e99b6f1891741e430e: No bootstrap required, opened a new log
I20260924 03:14:01.013826 18589 ts_tablet_manager.cc:1397] T 639f016d818d4ac2a6e45d0291d60154 P 7e43573e4ba341e99b6f1891741e430e: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260924 03:14:01.015097 18587 tablet_bootstrap.cc:492] T 639f016d818d4ac2a6e45d0291d60154 P d01839204681437abd135beb7616966e: No bootstrap required, opened a new log
I20260924 03:14:01.015439 18587 ts_tablet_manager.cc:1397] T 639f016d818d4ac2a6e45d0291d60154 P d01839204681437abd135beb7616966e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 03:14:01.018442 18586 raft_consensus.cc:348] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e43573e4ba341e99b6f1891741e430e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33599 } } peers { permanent_uuid: "41bceec5148848e8b6b576303ce5effb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46255 } } peers { permanent_uuid: "d01839204681437abd135beb7616966e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41475 } }
I20260924 03:14:01.018967 18586 raft_consensus.cc:374] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:01.019073 18586 raft_consensus.cc:729] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41bceec5148848e8b6b576303ce5effb, State: Initialized, Role: FOLLOWER
I20260924 03:14:01.019718 18586 consensus_queue.cc:260] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e43573e4ba341e99b6f1891741e430e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33599 } } peers { permanent_uuid: "41bceec5148848e8b6b576303ce5effb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46255 } } peers { permanent_uuid: "d01839204681437abd135beb7616966e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41475 } }
I20260924 03:14:01.020857 18568 heartbeater.cc:503] Master 127.0.97.62:42535 was elected leader, sending a full tablet report...
I20260924 03:14:01.021540 18586 ts_tablet_manager.cc:1428] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260924 03:14:01.024865 18589 raft_consensus.cc:348] T 639f016d818d4ac2a6e45d0291d60154 P 7e43573e4ba341e99b6f1891741e430e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e43573e4ba341e99b6f1891741e430e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33599 } } peers { permanent_uuid: "41bceec5148848e8b6b576303ce5effb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46255 } } peers { permanent_uuid: "d01839204681437abd135beb7616966e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41475 } }
I20260924 03:14:01.025360 18589 raft_consensus.cc:374] T 639f016d818d4ac2a6e45d0291d60154 P 7e43573e4ba341e99b6f1891741e430e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:01.025468 18589 raft_consensus.cc:729] T 639f016d818d4ac2a6e45d0291d60154 P 7e43573e4ba341e99b6f1891741e430e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e43573e4ba341e99b6f1891741e430e, State: Initialized, Role: FOLLOWER
I20260924 03:14:01.026131 18589 consensus_queue.cc:260] T 639f016d818d4ac2a6e45d0291d60154 P 7e43573e4ba341e99b6f1891741e430e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e43573e4ba341e99b6f1891741e430e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33599 } } peers { permanent_uuid: "41bceec5148848e8b6b576303ce5effb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46255 } } peers { permanent_uuid: "d01839204681437abd135beb7616966e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41475 } }
I20260924 03:14:01.026140 18587 raft_consensus.cc:348] T 639f016d818d4ac2a6e45d0291d60154 P d01839204681437abd135beb7616966e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e43573e4ba341e99b6f1891741e430e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33599 } } peers { permanent_uuid: "41bceec5148848e8b6b576303ce5effb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46255 } } peers { permanent_uuid: "d01839204681437abd135beb7616966e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41475 } }
I20260924 03:14:01.026597 18587 raft_consensus.cc:374] T 639f016d818d4ac2a6e45d0291d60154 P d01839204681437abd135beb7616966e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:01.026804 18587 raft_consensus.cc:729] T 639f016d818d4ac2a6e45d0291d60154 P d01839204681437abd135beb7616966e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d01839204681437abd135beb7616966e, State: Initialized, Role: FOLLOWER
I20260924 03:14:01.027531 18438 heartbeater.cc:503] Master 127.0.97.62:42535 was elected leader, sending a full tablet report...
I20260924 03:14:01.027451 18587 consensus_queue.cc:260] T 639f016d818d4ac2a6e45d0291d60154 P d01839204681437abd135beb7616966e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e43573e4ba341e99b6f1891741e430e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33599 } } peers { permanent_uuid: "41bceec5148848e8b6b576303ce5effb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46255 } } peers { permanent_uuid: "d01839204681437abd135beb7616966e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41475 } }
I20260924 03:14:01.028164 18589 ts_tablet_manager.cc:1428] T 639f016d818d4ac2a6e45d0291d60154 P 7e43573e4ba341e99b6f1891741e430e: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260924 03:14:01.029436 18587 ts_tablet_manager.cc:1428] T 639f016d818d4ac2a6e45d0291d60154 P d01839204681437abd135beb7616966e: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260924 03:14:01.076817 18592 raft_consensus.cc:482] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:14:01.077248 18592 raft_consensus.cc:504] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e43573e4ba341e99b6f1891741e430e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33599 } } peers { permanent_uuid: "41bceec5148848e8b6b576303ce5effb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46255 } } peers { permanent_uuid: "d01839204681437abd135beb7616966e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41475 } }
I20260924 03:14:01.079020 18592 leader_election.cc:288] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e43573e4ba341e99b6f1891741e430e (127.0.97.2:33599), d01839204681437abd135beb7616966e (127.0.97.1:41475)
I20260924 03:14:01.085579 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:01.079538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57457 (local address 127.0.97.1:41475)
0924 03:14:01.079723 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:01.079729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:01.079756 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:14:01.080045 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:01.080052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:01.080147 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:01.080263 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:01.080813 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:01.081703 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:01.083615 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:01.084600 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:01.084699 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:01.084777 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:01.085174 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 03:14:01.085325 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:14:01.085595 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:01.079333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41475 (local address 127.0.97.3:57457)
0924 03:14:01.079671 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:01.079696 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:01.079864 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:01.080370 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:01.080393 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:01.080424 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:01.080641 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:01.080655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:01.081846 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:01.081852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:01.083461 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:01.083473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:01.084762 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:01.084769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:01.084987 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:01.085023 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 03:14:01.085116 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 03:14:01.085249 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":44}
I20260924 03:14:01.085736 18595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:01.079331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33599 (local address 127.0.97.3:35905)
0924 03:14:01.080185 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:01.080204 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:01.080316 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:01.080716 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:01.080720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:01.080749 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:01.081129 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:01.081140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:01.082274 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:01.082277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:01.083741 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:01.083755 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:01.084964 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:01.084969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:01.085119 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:01.085154 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 03:14:01.085244 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 03:14:01.085363 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":62,"thread_start_us":474,"threads_started":1}
I20260924 03:14:01.085650 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:01.079450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35905 (local address 127.0.97.2:33599)
0924 03:14:01.079659 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:01.079666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:01.080292 (+ 626us) server_negotiation.cc:407] Connection header received
0924 03:14:01.080458 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:01.080466 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:01.080576 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:01.080678 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:01.081281 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:01.082126 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:01.083929 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:01.084802 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:01.084931 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:01.085032 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:01.085283 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 03:14:01.085420 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 03:14:01.086741 18262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "639f016d818d4ac2a6e45d0291d60154" candidate_uuid: "41bceec5148848e8b6b576303ce5effb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d01839204681437abd135beb7616966e" is_pre_election: true
I20260924 03:14:01.086746 18393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "639f016d818d4ac2a6e45d0291d60154" candidate_uuid: "41bceec5148848e8b6b576303ce5effb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e43573e4ba341e99b6f1891741e430e" is_pre_election: true
I20260924 03:14:01.087234 18393 raft_consensus.cc:2454] T 639f016d818d4ac2a6e45d0291d60154 P 7e43573e4ba341e99b6f1891741e430e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41bceec5148848e8b6b576303ce5effb in term 0.
I20260924 03:14:01.087234 18262 raft_consensus.cc:2454] T 639f016d818d4ac2a6e45d0291d60154 P d01839204681437abd135beb7616966e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41bceec5148848e8b6b576303ce5effb in term 0.
I20260924 03:14:01.087940 18459 leader_election.cc:302] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41bceec5148848e8b6b576303ce5effb, d01839204681437abd135beb7616966e; no voters:
I20260924 03:14:01.088367 18592 raft_consensus.cc:2781] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:14:01.088549 18592 raft_consensus.cc:482] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:14:01.088668 18592 raft_consensus.cc:3037] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:01.091109 18592 raft_consensus.cc:504] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e43573e4ba341e99b6f1891741e430e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33599 } } peers { permanent_uuid: "41bceec5148848e8b6b576303ce5effb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46255 } } peers { permanent_uuid: "d01839204681437abd135beb7616966e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41475 } }
I20260924 03:14:01.091830 18592 leader_election.cc:288] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb [CANDIDATE]: Term 1 election: Requested vote from peers 7e43573e4ba341e99b6f1891741e430e (127.0.97.2:33599), d01839204681437abd135beb7616966e (127.0.97.1:41475)
I20260924 03:14:01.092602 18262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "639f016d818d4ac2a6e45d0291d60154" candidate_uuid: "41bceec5148848e8b6b576303ce5effb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d01839204681437abd135beb7616966e"
I20260924 03:14:01.092602 18393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "639f016d818d4ac2a6e45d0291d60154" candidate_uuid: "41bceec5148848e8b6b576303ce5effb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e43573e4ba341e99b6f1891741e430e"
I20260924 03:14:01.092824 18393 raft_consensus.cc:3037] T 639f016d818d4ac2a6e45d0291d60154 P 7e43573e4ba341e99b6f1891741e430e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:01.092929 18262 raft_consensus.cc:3037] T 639f016d818d4ac2a6e45d0291d60154 P d01839204681437abd135beb7616966e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:01.096208 18262 raft_consensus.cc:2454] T 639f016d818d4ac2a6e45d0291d60154 P d01839204681437abd135beb7616966e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41bceec5148848e8b6b576303ce5effb in term 1.
I20260924 03:14:01.096259 18393 raft_consensus.cc:2454] T 639f016d818d4ac2a6e45d0291d60154 P 7e43573e4ba341e99b6f1891741e430e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41bceec5148848e8b6b576303ce5effb in term 1.
I20260924 03:14:01.096746 18459 leader_election.cc:302] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41bceec5148848e8b6b576303ce5effb, d01839204681437abd135beb7616966e; no voters:
I20260924 03:14:01.097111 18592 raft_consensus.cc:2781] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:01.097586 18592 raft_consensus.cc:686] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb [term 1 LEADER]: Becoming Leader. State: Replica: 41bceec5148848e8b6b576303ce5effb, State: Running, Role: LEADER
I20260924 03:14:01.098124 18592 consensus_queue.cc:237] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e43573e4ba341e99b6f1891741e430e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33599 } } peers { permanent_uuid: "41bceec5148848e8b6b576303ce5effb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46255 } } peers { permanent_uuid: "d01839204681437abd135beb7616966e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41475 } }
I20260924 03:14:01.102828 18120 catalog_manager.cc:5495] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb reported cstate change: term changed from 0 to 1, leader changed from <none> to 41bceec5148848e8b6b576303ce5effb (127.0.97.3). New cstate: current_term: 1 leader_uuid: "41bceec5148848e8b6b576303ce5effb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e43573e4ba341e99b6f1891741e430e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41bceec5148848e8b6b576303ce5effb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d01839204681437abd135beb7616966e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41475 } health_report { overall_health: UNKNOWN } } }
I20260924 03:14:01.119685 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:01.113651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.0.97.3:46255)
0924 03:14:01.113889 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:01.113893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:01.113911 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:14:01.114029 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:01.114034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:01.114120 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:01.114253 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:14:01.114781 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:01.115723 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:01.117272 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:01.117818 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:01.117940 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:01.118402 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:14:01.119010 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:14:01.119127 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:01.119296 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 03:14:01.119378 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260924 03:14:01.129014 18262 raft_consensus.cc:1261] T 639f016d818d4ac2a6e45d0291d60154 P d01839204681437abd135beb7616966e [term 1 FOLLOWER]: Refusing update from remote peer 41bceec5148848e8b6b576303ce5effb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:14:01.129014 18393 raft_consensus.cc:1261] T 639f016d818d4ac2a6e45d0291d60154 P 7e43573e4ba341e99b6f1891741e430e [term 1 FOLLOWER]: Refusing update from remote peer 41bceec5148848e8b6b576303ce5effb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:14:01.130314 18592 consensus_queue.cc:1035] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d01839204681437abd135beb7616966e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:14:01.130761 18596 consensus_queue.cc:1035] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e43573e4ba341e99b6f1891741e430e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 03:14:01.138710 18569 tablet.cc:2369] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:14:01.144099 18600 mvcc.cc:205] Tried to move back new op lower bound from 7332739650051723264 to 7332739649941843968. Current Snapshot: MvccSnapshot[applied={T|T < 7332739650051723264}]
I20260924 03:14:01.157809 18120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:14:01.158121 18120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:14:01.162536 18120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 795d98a846d440f790e5125d89004655: Sending DeleteTablet for 3 replicas of tablet 639f016d818d4ac2a6e45d0291d60154
I20260924 03:14:01.163902 18242 tablet_service.cc:1514] Processing DeleteTablet for tablet 639f016d818d4ac2a6e45d0291d60154 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:14:01 UTC) from {username='slave'} at 127.0.0.1:48380
I20260924 03:14:01.163904 18373 tablet_service.cc:1514] Processing DeleteTablet for tablet 639f016d818d4ac2a6e45d0291d60154 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:14:01 UTC) from {username='slave'} at 127.0.0.1:38526
I20260924 03:14:01.163932 18503 tablet_service.cc:1514] Processing DeleteTablet for tablet 639f016d818d4ac2a6e45d0291d60154 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:14:01 UTC) from {username='slave'} at 127.0.0.1:52228
I20260924 03:14:01.165117 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 18180
I20260924 03:14:01.182507 18608 tablet_replica.cc:330] T 639f016d818d4ac2a6e45d0291d60154 P d01839204681437abd135beb7616966e: stopping tablet replica
I20260924 03:14:01.183333 18608 raft_consensus.cc:2229] T 639f016d818d4ac2a6e45d0291d60154 P d01839204681437abd135beb7616966e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:14:01.184115 18608 raft_consensus.cc:2258] T 639f016d818d4ac2a6e45d0291d60154 P d01839204681437abd135beb7616966e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:14:01.190198 18608 ts_tablet_manager.cc:1905] T 639f016d818d4ac2a6e45d0291d60154 P d01839204681437abd135beb7616966e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:14:01.191026 18609 tablet_replica.cc:330] T 639f016d818d4ac2a6e45d0291d60154 P 7e43573e4ba341e99b6f1891741e430e: stopping tablet replica
I20260924 03:14:01.192020 18610 tablet_replica.cc:330] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb: stopping tablet replica
I20260924 03:14:01.192960 18610 raft_consensus.cc:2229] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:14:01.193748 18610 raft_consensus.cc:2258] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:14:01.194260 18609 raft_consensus.cc:2229] T 639f016d818d4ac2a6e45d0291d60154 P 7e43573e4ba341e99b6f1891741e430e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:14:01.194902 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:01.166953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48388 (local address 127.0.97.1:41475)
0924 03:14:01.167202 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:01.167211 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:01.177582 (+ 10371us) server_negotiation.cc:407] Connection header received
0924 03:14:01.178209 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:01.178219 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:01.178382 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:01.178480 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:01.179362 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:01.180902 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:01.187354 (+ 6452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:01.188251 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:01.188387 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:01.192971 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:01.192987 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:01.193005 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:01.193857 (+ 852us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:01.193939 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:01.193968 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:01.194083 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:01.194334 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 03:14:01.194459 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 03:14:01.194890 18609 raft_consensus.cc:2258] T 639f016d818d4ac2a6e45d0291d60154 P 7e43573e4ba341e99b6f1891741e430e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:14:01.196066 18302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:01.197151 18610 ts_tablet_manager.cc:1905] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:14:01.200587 18609 ts_tablet_manager.cc:1905] T 639f016d818d4ac2a6e45d0291d60154 P 7e43573e4ba341e99b6f1891741e430e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:14:01.205220 18610 ts_tablet_manager.cc:1918] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:14:01.205503 18610 log.cc:1196] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/639f016d818d4ac2a6e45d0291d60154
I20260924 03:14:01.206151 18610 ts_tablet_manager.cc:1939] T 639f016d818d4ac2a6e45d0291d60154 P 41bceec5148848e8b6b576303ce5effb: Deleting consensus metadata
I20260924 03:14:01.207795 18609 ts_tablet_manager.cc:1918] T 639f016d818d4ac2a6e45d0291d60154 P 7e43573e4ba341e99b6f1891741e430e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:14:01.208035 18609 log.cc:1196] T 639f016d818d4ac2a6e45d0291d60154 P 7e43573e4ba341e99b6f1891741e430e: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/639f016d818d4ac2a6e45d0291d60154
I20260924 03:14:01.208388 18108 catalog_manager.cc:4841] TS 41bceec5148848e8b6b576303ce5effb (127.0.97.3:46255): tablet 639f016d818d4ac2a6e45d0291d60154 (table test-table [id=de9e689008714fd8a449f9dfa11e6cc2]) successfully deleted
I20260924 03:14:01.208750 18609 ts_tablet_manager.cc:1939] T 639f016d818d4ac2a6e45d0291d60154 P 7e43573e4ba341e99b6f1891741e430e: Deleting consensus metadata
I20260924 03:14:01.210716 18107 catalog_manager.cc:4841] TS 7e43573e4ba341e99b6f1891741e430e (127.0.97.2:33599): tablet 639f016d818d4ac2a6e45d0291d60154 (table test-table [id=de9e689008714fd8a449f9dfa11e6cc2]) successfully deleted
I20260924 03:14:01.290424 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 18180
W20260924 03:14:01.307965 18108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:41475 (error 108)
I20260924 03:14:01.308802 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 18311
I20260924 03:14:01.310230 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:01.309722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41475 (local address 127.0.0.1:48402)
0924 03:14:01.309924 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:01.310054 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 03:14:01.310621 18108 catalog_manager.cc:4584] TS d01839204681437abd135beb7616966e (127.0.97.1:41475): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 639f016d818d4ac2a6e45d0291d60154: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41475: connect: Connection refused (error 111)
I20260924 03:14:01.317200 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:01.309685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38528 (local address 127.0.97.2:33599)
0924 03:14:01.310017 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:01.310026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:01.310046 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:14:01.310120 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:01.310124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:01.310220 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:01.310352 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:01.310803 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:01.311952 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:01.313308 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:01.313971 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:01.314105 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:01.315274 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:01.315296 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:01.315316 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:01.316249 (+ 933us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:01.316326 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:01.316353 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:01.316479 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:01.316711 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 03:14:01.316839 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260924 03:14:01.318010 18433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:01.364593 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:01.364037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41475 (local address 127.0.0.1:48412)
0924 03:14:01.364239 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:01.364338 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 03:14:01.365949 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:01.365510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41475 (local address 127.0.0.1:48418)
0924 03:14:01.365711 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:01.365792 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260924 03:14:01.412997 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 18311
I20260924 03:14:01.426945 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:01.426437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41475 (local address 127.0.0.1:48434)
0924 03:14:01.426667 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:01.426754 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 03:14:01.428033 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:01.427682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41475 (local address 127.0.0.1:48444)
0924 03:14:01.427849 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:01.427934 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 03:14:01.435941 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 18441
I20260924 03:14:01.444119 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:01.436917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52244 (local address 127.0.97.3:46255)
0924 03:14:01.437114 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:01.437185 (+ 71us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:01.437204 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:14:01.437267 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:01.437271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:01.437353 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:01.437457 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:01.437943 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:01.438869 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:01.440879 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:01.441339 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:01.441462 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:01.442665 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:01.442672 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:01.442686 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:01.443296 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:01.443348 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:01.443366 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:01.443451 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:01.443621 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 03:14:01.443704 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 03:14:01.444927 18563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:01.509117 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:01.508379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41475 (local address 127.0.0.1:48456)
0924 03:14:01.508837 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:01.508950 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
I20260924 03:14:01.510284 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:01.509874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41475 (local address 127.0.0.1:48464)
0924 03:14:01.510081 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:01.510160 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260924 03:14:01.566977 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 18441
I20260924 03:14:01.586973 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 18090
I20260924 03:14:01.588268 18151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:01.700872 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:01.699894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41475 (local address 127.0.0.1:48478)
0924 03:14:01.700472 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:01.700627 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332}
I20260924 03:14:01.700896 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 18090
2026-09-24T03:14:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2889 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:01 dist-test-slave-5l98 krb5kdc[18619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:01 dist-test-slave-5l98 krb5kdc[18619](info): set up 2 sockets
Sep 24 03:14:01 dist-test-slave-5l98 krb5kdc[18619](info): commencing operation
krb5kdc: starting...
W20260924 03:14:02.840835 388 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:03 dist-test-slave-5l98 krb5kdc[18619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:14:03.106415 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35383
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:03.443704 18632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:03.444074 18632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:03.444247 18632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:03.451612 18632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:03.451778 18632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:03.451887 18632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:03.451994 18632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:03.452077 18632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:03.462232 18632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:03.463667 18632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:03.465014 18632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:03.477147 18637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:03.477187 18638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:03.477077 18640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:03.478235 18632 server_base.cc:1034] running on GCE node
I20260924 03:14:03.478999 18632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:03.480430 18632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:03.481781 18632 hybrid_clock.cc:648] HybridClock initialized: now 1790219643481686 us; error 88 us; skew 500 ppm
I20260924 03:14:03.482167 18632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:03 dist-test-slave-5l98 krb5kdc[18619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:03.485412 18632 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:14:03.486946 18632 webserver.cc:460] Webserver started at http://127.0.97.62:44401/ using document root <none> and password file <none>
I20260924 03:14:03.487584 18632 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:03.487694 18632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:03.487946 18632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:03.491012 18632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "3fd6386a88b94eecb098601594855435"
format_stamp: "Formatted at 2026-09-24 03:14:03 on dist-test-slave-5l98"
server_key: "395f2eb6d3c29a5668c977d02c5096c1"
server_key_iv: "f093e8b99762b2a3acd66c92ed60b428"
server_key_version: "encryptionkey@0"
I20260924 03:14:03.491721 18632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "3fd6386a88b94eecb098601594855435"
format_stamp: "Formatted at 2026-09-24 03:14:03 on dist-test-slave-5l98"
server_key: "395f2eb6d3c29a5668c977d02c5096c1"
server_key_iv: "f093e8b99762b2a3acd66c92ed60b428"
server_key_version: "encryptionkey@0"
I20260924 03:14:03.496944 18632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260924 03:14:03.501341 18647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:03.502866 18632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260924 03:14:03.503068 18632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "3fd6386a88b94eecb098601594855435"
format_stamp: "Formatted at 2026-09-24 03:14:03 on dist-test-slave-5l98"
server_key: "395f2eb6d3c29a5668c977d02c5096c1"
server_key_iv: "f093e8b99762b2a3acd66c92ed60b428"
server_key_version: "encryptionkey@0"
I20260924 03:14:03.503273 18632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:03.522562 18632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:03.526921 18632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:03.527235 18632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:03.547214 18632 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35383
I20260924 03:14:03.547302 18698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35383 every 8 connection(s)
I20260924 03:14:03.548667 18632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:03.551776 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 18632
I20260924 03:14:03.551970 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:03.552448 388 external_mini_cluster.cc:1428] Setting key 1375049cf9e8b07c42e35dfa067abceb
I20260924 03:14:03.552767 18699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:03.568351 18699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435: Bootstrap starting.
Sep 24 03:14:03 dist-test-slave-5l98 krb5kdc[18619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:14:03.572209 18699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:03.573441 18699 log.cc:824] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:03.576436 18699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435: No bootstrap required, opened a new log
I20260924 03:14:03.577955 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:03.554353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35552 (local address 127.0.97.62:35383)
0924 03:14:03.555202 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:03.555213 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:03.555260 (+ 47us) server_negotiation.cc:407] Connection header received
0924 03:14:03.557624 (+ 2364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:03.557647 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:03.558176 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:03.558518 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:03.560440 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:03.561891 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:03.563832 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:03.564662 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:03.564819 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:03.570247 (+ 5428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:03.570256 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:03.570275 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:03.570318 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:03.573727 (+ 3409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:03.574529 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:03.574545 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:03.574551 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:03.574732 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:03.575154 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:03.575160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:03.575164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:03.575714 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:03.576032 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:03.576519 (+ 487us) server_negotiation.cc:299] Negotiation successful
0924 03:14:03.576983 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":400,"threads_started":1}
I20260924 03:14:03.585153 18699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd6386a88b94eecb098601594855435" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } }
I20260924 03:14:03.585634 18699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:03.585750 18699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd6386a88b94eecb098601594855435, State: Initialized, Role: FOLLOWER
I20260924 03:14:03.586319 18699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd6386a88b94eecb098601594855435" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } }
I20260924 03:14:03.586606 18699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:03.586762 18699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:03.586990 18699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:03.589593 18699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd6386a88b94eecb098601594855435" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } }
I20260924 03:14:03.590171 18699 leader_election.cc:302] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fd6386a88b94eecb098601594855435; no voters:
I20260924 03:14:03.590867 18699 leader_election.cc:288] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:03.590975 18704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:03.591668 18704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435 [term 1 LEADER]: Becoming Leader. State: Replica: 3fd6386a88b94eecb098601594855435, State: Running, Role: LEADER
I20260924 03:14:03.592317 18704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd6386a88b94eecb098601594855435" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } }
I20260924 03:14:03.593300 18699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:03.595315 18705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fd6386a88b94eecb098601594855435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd6386a88b94eecb098601594855435" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } }
I20260924 03:14:03.595767 18705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:03.596300 18706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fd6386a88b94eecb098601594855435. Latest consensus state: current_term: 1 leader_uuid: "3fd6386a88b94eecb098601594855435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd6386a88b94eecb098601594855435" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } }
I20260924 03:14:03.596650 18706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:03.597594 18710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:03.602890 18710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:03.612200 18710 catalog_manager.cc:1347] Generated new cluster ID: 53f520885d6a4c8390a7c0bd20453a40
I20260924 03:14:03.612327 18710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:03.623347 18710 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:03.624763 18710 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:03.636106 18710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435: Generated new TSK 0
I20260924 03:14:03.637041 18710 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 03:14:03.710208 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35383
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:04.067750 18725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:04.068046 18725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:04.068143 18725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:04.075598 18725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:04.075702 18725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:04.075819 18725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:14:04.085763 18725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:04.087019 18725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:04.088406 18725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:04.103216 18733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:04.103248 18731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:04.103336 18730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:04.103891 18725 server_base.cc:1034] running on GCE node
I20260924 03:14:04.104624 18725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:04.105536 18725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:04.106750 18725 hybrid_clock.cc:648] HybridClock initialized: now 1790219644106655 us; error 87 us; skew 500 ppm
I20260924 03:14:04.107191 18725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:04 dist-test-slave-5l98 krb5kdc[18619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:04.110448 18725 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:14:04.112103 18725 webserver.cc:460] Webserver started at http://127.0.97.1:35169/ using document root <none> and password file <none>
I20260924 03:14:04.112912 18725 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:04.113062 18725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:04.113391 18725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:04.116272 18725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "0ad24e75ec944c728b5bdfadf346777e"
format_stamp: "Formatted at 2026-09-24 03:14:04 on dist-test-slave-5l98"
server_key: "1466e4dc1ff6853fbcec75f586faeabc"
server_key_iv: "2779600113caacf24fd505609b258306"
server_key_version: "encryptionkey@0"
I20260924 03:14:04.117127 18725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0ad24e75ec944c728b5bdfadf346777e"
format_stamp: "Formatted at 2026-09-24 03:14:04 on dist-test-slave-5l98"
server_key: "1466e4dc1ff6853fbcec75f586faeabc"
server_key_iv: "2779600113caacf24fd505609b258306"
server_key_version: "encryptionkey@0"
I20260924 03:14:04.122900 18725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:14:04.126573 18740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:04.127880 18725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:14:04.128062 18725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "0ad24e75ec944c728b5bdfadf346777e"
format_stamp: "Formatted at 2026-09-24 03:14:04 on dist-test-slave-5l98"
server_key: "1466e4dc1ff6853fbcec75f586faeabc"
server_key_iv: "2779600113caacf24fd505609b258306"
server_key_version: "encryptionkey@0"
I20260924 03:14:04.128276 18725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:04.172215 18725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:04.176874 18725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:04.177194 18725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:04.178256 18725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:14:04.179683 18725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:14:04.179796 18725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:04.179942 18725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:14:04.180013 18725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:04.218062 18725 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38347
I20260924 03:14:04.218107 18852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38347 every 8 connection(s)
I20260924 03:14:04.219410 18725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:14:04.220412 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 18725
I20260924 03:14:04.220623 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:14:04.221172 388 external_mini_cluster.cc:1428] Setting key 3e4ccef635dcaf1596c65fdfacd0c096
Sep 24 03:14:04 dist-test-slave-5l98 krb5kdc[18619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:14:04.243055 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:04.222242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37633 (local address 127.0.97.62:35383)
0924 03:14:04.222761 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:04.222767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:04.223835 (+ 1068us) server_negotiation.cc:407] Connection header received
0924 03:14:04.225766 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:04.225774 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:04.225865 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:04.225983 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:04.227643 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:04.228916 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:04.230629 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:04.231135 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:04.231291 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:04.235606 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:04.235615 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:04.235622 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:04.235671 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:04.239366 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:04.240217 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:04.240233 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:04.240239 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:04.240399 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:04.240899 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:04.240908 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:04.240914 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:04.241314 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:04.241480 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:04.242263 (+ 783us) server_negotiation.cc:299] Negotiation successful
0924 03:14:04.242548 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":245,"threads_started":1}
I20260924 03:14:04.243831 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:04.222860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.1:37633)
0924 03:14:04.223663 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:04.223715 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:04.225488 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:04.226179 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:04.226187 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:04.226831 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:04.227454 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:04.227474 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:04.229101 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:04.229105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:04.230471 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:04.230483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:04.231387 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:04.231396 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:04.231702 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:04.232599 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:14:04.232621 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:04.235364 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:04.239568 (+ 4204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:04.239576 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:04.239589 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:04.240081 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:04.240582 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:04.240585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:04.240588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:04.240783 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:04.241466 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:04.241472 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:04.241833 (+ 361us) client_negotiation.cc:769] Sending connection context
0924 03:14:04.242071 (+ 238us) client_negotiation.cc:240] Negotiation successful
0924 03:14:04.242573 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":348,"threads_started":1}
I20260924 03:14:04.245724 18853 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35383
I20260924 03:14:04.246101 18853 heartbeater.cc:464] Registering TS with master...
I20260924 03:14:04.247038 18853 heartbeater.cc:511] Master 127.0.97.62:35383 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 03:14:04.249532 18664 ts_manager.cc:194] Registered new tserver with Master: 0ad24e75ec944c728b5bdfadf346777e (127.0.97.1:38347)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 03:14:04.251384 18664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 03:14:04.284781 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35383
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:04.618702 18859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:04.618988 18859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:04.619127 18859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:04.626799 18859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:04.626914 18859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:04.627084 18859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:14:04.637181 18859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:04.638481 18859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:04.639760 18859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:04.653319 18864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:04.653264 18867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:04.653324 18865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:04.658124 18859 server_base.cc:1034] running on GCE node
I20260924 03:14:04.658825 18859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:04.659965 18859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:04.661294 18859 hybrid_clock.cc:648] HybridClock initialized: now 1790219644661250 us; error 32 us; skew 500 ppm
I20260924 03:14:04.661697 18859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:04 dist-test-slave-5l98 krb5kdc[18619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:04.671417 18859 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:14:04.673317 18859 webserver.cc:460] Webserver started at http://127.0.97.2:34029/ using document root <none> and password file <none>
I20260924 03:14:04.674154 18859 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:04.674273 18859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:04.674547 18859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:04.677611 18859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "7ec44523932b460f973ea6c2e1fc8f74"
format_stamp: "Formatted at 2026-09-24 03:14:04 on dist-test-slave-5l98"
server_key: "b048d00dd63bbc59f4760b1108b9f7d0"
server_key_iv: "c9cd1bdd7f9bbb7dd7010d97a808193f"
server_key_version: "encryptionkey@0"
I20260924 03:14:04.678378 18859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7ec44523932b460f973ea6c2e1fc8f74"
format_stamp: "Formatted at 2026-09-24 03:14:04 on dist-test-slave-5l98"
server_key: "b048d00dd63bbc59f4760b1108b9f7d0"
server_key_iv: "c9cd1bdd7f9bbb7dd7010d97a808193f"
server_key_version: "encryptionkey@0"
I20260924 03:14:04.683712 18859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:14:04.687575 18874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:04.689091 18859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:14:04.689296 18859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "7ec44523932b460f973ea6c2e1fc8f74"
format_stamp: "Formatted at 2026-09-24 03:14:04 on dist-test-slave-5l98"
server_key: "b048d00dd63bbc59f4760b1108b9f7d0"
server_key_iv: "c9cd1bdd7f9bbb7dd7010d97a808193f"
server_key_version: "encryptionkey@0"
I20260924 03:14:04.689514 18859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:04.719666 18859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:04.723867 18859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:04.724182 18859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:04.725203 18859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:14:04.726557 18859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:14:04.726660 18859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:04.726809 18859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:14:04.726888 18859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:04.761291 18859 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42831
I20260924 03:14:04.761410 18986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42831 every 8 connection(s)
I20260924 03:14:04.762594 18859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:14:04.770465 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 18859
I20260924 03:14:04.770692 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:14:04.771123 388 external_mini_cluster.cc:1428] Setting key 9a62fa27fc119673de5c213b2293ddfa
Sep 24 03:14:04 dist-test-slave-5l98 krb5kdc[18619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:14:04.785005 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:04.765085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46293 (local address 127.0.97.62:35383)
0924 03:14:04.765548 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:04.765555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:04.766495 (+ 940us) server_negotiation.cc:407] Connection header received
0924 03:14:04.768390 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:04.768394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:04.768468 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:04.768727 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:04.770542 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:04.771420 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:04.773489 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:04.773945 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:04.774050 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:04.778549 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:04.778554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:04.778558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:04.778594 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:04.781493 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:04.782277 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:04.782295 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:04.782297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:04.782424 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:04.782938 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:04.782945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:04.782948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:04.783237 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:04.783595 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:04.784342 (+ 747us) server_negotiation.cc:299] Negotiation successful
0924 03:14:04.784602 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":210,"threads_started":1}
I20260924 03:14:04.786005 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:04.765581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.2:46293)
0924 03:14:04.766336 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:04.766372 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:04.768068 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:04.769081 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:04.769090 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:04.769620 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:04.770339 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:04.770363 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:04.771594 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:04.771599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:04.773321 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:04.773336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:04.774191 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:04.774200 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:04.774561 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:04.775365 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:14:04.775400 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:04.778280 (+ 2880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:04.781667 (+ 3387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:04.781678 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:04.781705 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:04.782140 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:04.782552 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:04.782557 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:04.782561 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:04.782826 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:04.783555 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:04.783563 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:04.783960 (+ 397us) client_negotiation.cc:769] Sending connection context
0924 03:14:04.784250 (+ 290us) client_negotiation.cc:240] Negotiation successful
0924 03:14:04.784850 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":332,"threads_started":1}
I20260924 03:14:04.787631 18987 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35383
I20260924 03:14:04.787988 18987 heartbeater.cc:464] Registering TS with master...
I20260924 03:14:04.788959 18987 heartbeater.cc:511] Master 127.0.97.62:35383 requested a full tablet report, sending...
I20260924 03:14:04.790796 18664 ts_manager.cc:194] Registered new tserver with Master: 7ec44523932b460f973ea6c2e1fc8f74 (127.0.97.2:42831)
I20260924 03:14:04.792230 18664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46293
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 03:14:04.833878 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35383
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:05.166353 18993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:05.166630 18993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:05.166802 18993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:05.174739 18993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:05.174861 18993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:05.175081 18993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:14:05.184883 18993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:05.186182 18993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:05.187662 18993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:05.202842 18998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:05.202762 18999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:05.202737 19001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:05.203125 18993 server_base.cc:1034] running on GCE node
I20260924 03:14:05.204090 18993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:05.205315 18993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:05.206604 18993 hybrid_clock.cc:648] HybridClock initialized: now 1790219645206552 us; error 76 us; skew 500 ppm
I20260924 03:14:05.206956 18993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:05 dist-test-slave-5l98 krb5kdc[18619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:05.210472 18993 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:14:05.211985 18993 webserver.cc:460] Webserver started at http://127.0.97.3:36905/ using document root <none> and password file <none>
I20260924 03:14:05.212708 18993 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:05.212819 18993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:05.213081 18993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:05.215960 18993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "290292b8a444429aa96648fca407cd30"
format_stamp: "Formatted at 2026-09-24 03:14:05 on dist-test-slave-5l98"
server_key: "1db3a6645968c9ae24221657d6183e21"
server_key_iv: "93ed76cc59fe5918a75cad894c4ce06f"
server_key_version: "encryptionkey@0"
I20260924 03:14:05.216761 18993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "290292b8a444429aa96648fca407cd30"
format_stamp: "Formatted at 2026-09-24 03:14:05 on dist-test-slave-5l98"
server_key: "1db3a6645968c9ae24221657d6183e21"
server_key_iv: "93ed76cc59fe5918a75cad894c4ce06f"
server_key_version: "encryptionkey@0"
I20260924 03:14:05.222337 18993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:14:05.225950 19008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:05.227536 18993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:14:05.227721 18993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "290292b8a444429aa96648fca407cd30"
format_stamp: "Formatted at 2026-09-24 03:14:05 on dist-test-slave-5l98"
server_key: "1db3a6645968c9ae24221657d6183e21"
server_key_iv: "93ed76cc59fe5918a75cad894c4ce06f"
server_key_version: "encryptionkey@0"
I20260924 03:14:05.227977 18993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:05.248271 18993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:05.253170 18993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:05.253501 18993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:05.254630 18993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:14:05.254606 18853 heartbeater.cc:503] Master 127.0.97.62:35383 was elected leader, sending a full tablet report...
I20260924 03:14:05.256340 18993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:14:05.256464 18993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:05.256721 18993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:14:05.256817 18993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:05.293781 18993 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45741
I20260924 03:14:05.293869 19120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45741 every 8 connection(s)
I20260924 03:14:05.295203 18993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:14:05.299692 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 18993
I20260924 03:14:05.299883 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:14:05.300472 388 external_mini_cluster.cc:1428] Setting key 37998c4e7342e3840e083c7dfc32140b
Sep 24 03:14:05 dist-test-slave-5l98 krb5kdc[18619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:14:05.317158 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:05.297869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48111 (local address 127.0.97.62:35383)
0924 03:14:05.298473 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:05.298477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:05.299246 (+ 769us) server_negotiation.cc:407] Connection header received
0924 03:14:05.301777 (+ 2531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:05.301785 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:05.301947 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:05.302042 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:05.304138 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.304985 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.306652 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.307112 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.307213 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:05.310820 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:05.310825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:05.310829 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:05.310861 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:05.313641 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:05.314465 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:05.314473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:05.314479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:05.314610 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:05.315167 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:05.315171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:05.315173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:05.315600 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:05.315771 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:05.316596 (+ 825us) server_negotiation.cc:299] Negotiation successful
0924 03:14:05.316867 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":375,"threads_started":1}
I20260924 03:14:05.317973 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:05.298238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.3:48111)
0924 03:14:05.299098 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:05.299135 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:05.301387 (+ 2252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:05.302324 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:05.302332 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:05.303144 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:05.303831 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:05.303851 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.305136 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.305141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:05.306522 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:05.306534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.307307 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.307312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:05.307562 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:05.308270 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:14:05.308297 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:05.310530 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:05.313802 (+ 3272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:05.313810 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:05.313826 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:05.314334 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:05.314726 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:05.314730 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:05.314734 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:05.315025 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:05.315747 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:05.315753 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:05.316119 (+ 366us) client_negotiation.cc:769] Sending connection context
0924 03:14:05.316427 (+ 308us) client_negotiation.cc:240] Negotiation successful
0924 03:14:05.316884 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":363,"threads_started":1}
I20260924 03:14:05.319437 19121 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35383
I20260924 03:14:05.319720 19121 heartbeater.cc:464] Registering TS with master...
I20260924 03:14:05.320401 19121 heartbeater.cc:511] Master 127.0.97.62:35383 requested a full tablet report, sending...
I20260924 03:14:05.321851 18664 ts_manager.cc:194] Registered new tserver with Master: 290292b8a444429aa96648fca407cd30 (127.0.97.3:45741)
I20260924 03:14:05.322649 18664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48111
I20260924 03:14:05.328316 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:14:05.346364 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:05.332559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35562 (local address 127.0.97.62:35383)
0924 03:14:05.332741 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:05.332745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:05.333169 (+ 424us) server_negotiation.cc:407] Connection header received
0924 03:14:05.333318 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:05.333322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:05.333404 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:05.333482 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:05.334947 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.335628 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.337233 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.337648 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.337785 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:05.340702 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:05.340709 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:05.340714 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:05.340753 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:05.343373 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:05.344217 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:05.344222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:05.344224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:05.344351 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:05.344857 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:05.344861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:05.344864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:05.345175 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:05.345489 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:05.345856 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 03:14:05.346052 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:14:05.350848 18664 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:14:05.360576 18664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:14:05.387635 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:05.373915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45741 (local address 127.0.0.1:58770)
0924 03:14:05.375972 (+ 2057us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:05.376028 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:05.376227 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:05.377833 (+ 1606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:05.377842 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:05.377888 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:05.378363 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:05.378378 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.381197 (+ 2819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.381202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:05.384071 (+ 2869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:05.384093 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.386269 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.386280 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:05.386560 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:05.386639 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 03:14:05.386774 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 03:14:05.387051 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1727,"thread_start_us":3019,"threads_started":1}
I20260924 03:14:05.389065 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:05.374073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58770 (local address 127.0.97.3:45741)
0924 03:14:05.376961 (+ 2888us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:05.376972 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:05.377011 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:14:05.377107 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:05.377116 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:05.377600 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:05.378545 (+ 945us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:05.378816 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.381024 (+ 2208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.384345 (+ 3321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.386078 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.388145 (+ 2067us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:05.388415 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:05.388508 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 03:14:05.388700 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2696,"thread_start_us":2529,"threads_started":1}
I20260924 03:14:05.393270 19056 tablet_service.cc:1467] Processing CreateTablet for tablet 270e7cc6aa764b658403f426072c6f7d (DEFAULT_TABLE table=test-table [id=69a89715991847099399a039d0aef5b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:14:05.394910 19056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270e7cc6aa764b658403f426072c6f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:05.396306 19135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:05.374009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42831 (local address 127.0.0.1:49380)
0924 03:14:05.377200 (+ 3191us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:05.377223 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:05.377330 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:05.388759 (+ 11429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:05.388766 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:05.388795 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:05.389146 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:05.389157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.391193 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.391198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:05.393212 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:05.393225 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.395201 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.395207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:05.395401 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:05.395448 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 03:14:05.395513 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 03:14:05.395651 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3078,"mutex_wait_us":148,"thread_start_us":1465,"threads_started":1}
I20260924 03:14:05.396898 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:05.378660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38347 (local address 127.0.0.1:48420)
0924 03:14:05.387880 (+ 9220us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:05.387899 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:05.387990 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:05.389443 (+ 1453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:05.389448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:05.389472 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:05.389685 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:05.389696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.391540 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.391545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:05.393583 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:05.393596 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.395987 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.395994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:05.396188 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:05.396222 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:14:05.396351 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 03:14:05.396539 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9117,"mutex_wait_us":25,"thread_start_us":9564,"threads_started":1}
I20260924 03:14:05.397497 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:05.374654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49380 (local address 127.0.97.2:42831)
0924 03:14:05.387641 (+ 12987us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:05.387652 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:05.387681 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:14:05.387792 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:05.387802 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:05.388069 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:05.388750 (+ 681us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:05.389384 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.391043 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.393409 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.394976 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.396704 (+ 1728us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:05.396907 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:05.396993 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 03:14:05.397176 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12815,"thread_start_us":12641,"threads_started":1}
I20260924 03:14:05.397879 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:05.378491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.0.97.1:38347)
0924 03:14:05.388796 (+ 10305us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:05.388805 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:05.388834 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:14:05.388945 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:05.388951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:05.389183 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:05.389427 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:05.389844 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.391370 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.393786 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.395351 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.395468 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:05.395692 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:05.397228 (+ 1536us) server_negotiation.cc:299] Negotiation successful
0924 03:14:05.397484 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10099,"thread_start_us":9926,"threads_started":1}
I20260924 03:14:05.400967 18922 tablet_service.cc:1467] Processing CreateTablet for tablet 270e7cc6aa764b658403f426072c6f7d (DEFAULT_TABLE table=test-table [id=69a89715991847099399a039d0aef5b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:14:05.401516 18788 tablet_service.cc:1467] Processing CreateTablet for tablet 270e7cc6aa764b658403f426072c6f7d (DEFAULT_TABLE table=test-table [id=69a89715991847099399a039d0aef5b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:14:05.402232 18922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270e7cc6aa764b658403f426072c6f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:05.403085 18788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270e7cc6aa764b658403f426072c6f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:05.411034 19140 tablet_bootstrap.cc:492] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30: Bootstrap starting.
I20260924 03:14:05.415351 19140 tablet_bootstrap.cc:654] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:05.415915 19141 tablet_bootstrap.cc:492] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74: Bootstrap starting.
I20260924 03:14:05.416869 19140 log.cc:824] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:05.419654 19140 tablet_bootstrap.cc:492] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30: No bootstrap required, opened a new log
I20260924 03:14:05.419978 19140 ts_tablet_manager.cc:1397] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 03:14:05.420060 19141 tablet_bootstrap.cc:654] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:05.420482 19143 tablet_bootstrap.cc:492] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e: Bootstrap starting.
I20260924 03:14:05.421361 19141 log.cc:824] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:05.424132 19141 tablet_bootstrap.cc:492] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74: No bootstrap required, opened a new log
I20260924 03:14:05.424363 19143 tablet_bootstrap.cc:654] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:05.424510 19141 ts_tablet_manager.cc:1397] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260924 03:14:05.425724 19143 log.cc:824] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:05.429143 19143 tablet_bootstrap.cc:492] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e: No bootstrap required, opened a new log
I20260924 03:14:05.429518 19143 ts_tablet_manager.cc:1397] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 03:14:05.431231 19140 raft_consensus.cc:348] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290292b8a444429aa96648fca407cd30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45741 } } peers { permanent_uuid: "7ec44523932b460f973ea6c2e1fc8f74" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42831 } } peers { permanent_uuid: "0ad24e75ec944c728b5bdfadf346777e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38347 } }
I20260924 03:14:05.431731 19140 raft_consensus.cc:374] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:05.431905 19140 raft_consensus.cc:729] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290292b8a444429aa96648fca407cd30, State: Initialized, Role: FOLLOWER
I20260924 03:14:05.432708 19140 consensus_queue.cc:260] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290292b8a444429aa96648fca407cd30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45741 } } peers { permanent_uuid: "7ec44523932b460f973ea6c2e1fc8f74" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42831 } } peers { permanent_uuid: "0ad24e75ec944c728b5bdfadf346777e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38347 } }
I20260924 03:14:05.433908 19121 heartbeater.cc:503] Master 127.0.97.62:35383 was elected leader, sending a full tablet report...
I20260924 03:14:05.433934 19141 raft_consensus.cc:348] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290292b8a444429aa96648fca407cd30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45741 } } peers { permanent_uuid: "7ec44523932b460f973ea6c2e1fc8f74" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42831 } } peers { permanent_uuid: "0ad24e75ec944c728b5bdfadf346777e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38347 } }
I20260924 03:14:05.434350 19141 raft_consensus.cc:374] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:05.434469 19141 raft_consensus.cc:729] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec44523932b460f973ea6c2e1fc8f74, State: Initialized, Role: FOLLOWER
I20260924 03:14:05.435067 19141 consensus_queue.cc:260] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290292b8a444429aa96648fca407cd30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45741 } } peers { permanent_uuid: "7ec44523932b460f973ea6c2e1fc8f74" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42831 } } peers { permanent_uuid: "0ad24e75ec944c728b5bdfadf346777e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38347 } }
I20260924 03:14:05.435961 19140 ts_tablet_manager.cc:1428] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30: Time spent starting tablet: real 0.016s user 0.006s sys 0.008s
I20260924 03:14:05.439626 18987 heartbeater.cc:503] Master 127.0.97.62:35383 was elected leader, sending a full tablet report...
I20260924 03:14:05.440320 19141 ts_tablet_manager.cc:1428] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260924 03:14:05.440111 19143 raft_consensus.cc:348] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290292b8a444429aa96648fca407cd30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45741 } } peers { permanent_uuid: "7ec44523932b460f973ea6c2e1fc8f74" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42831 } } peers { permanent_uuid: "0ad24e75ec944c728b5bdfadf346777e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38347 } }
I20260924 03:14:05.440601 19143 raft_consensus.cc:374] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:05.440694 19143 raft_consensus.cc:729] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ad24e75ec944c728b5bdfadf346777e, State: Initialized, Role: FOLLOWER
I20260924 03:14:05.441283 19143 consensus_queue.cc:260] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290292b8a444429aa96648fca407cd30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45741 } } peers { permanent_uuid: "7ec44523932b460f973ea6c2e1fc8f74" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42831 } } peers { permanent_uuid: "0ad24e75ec944c728b5bdfadf346777e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38347 } }
I20260924 03:14:05.443081 19143 ts_tablet_manager.cc:1428] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260924 03:14:05.472204 18854 tablet.cc:2369] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:14:05.514773 18988 tablet.cc:2369] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:14:05.547087 19122 tablet.cc:2369] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:14:05.548424 19146 raft_consensus.cc:482] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:14:05.548884 19146 raft_consensus.cc:504] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290292b8a444429aa96648fca407cd30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45741 } } peers { permanent_uuid: "7ec44523932b460f973ea6c2e1fc8f74" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42831 } } peers { permanent_uuid: "0ad24e75ec944c728b5bdfadf346777e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38347 } }
I20260924 03:14:05.550599 19146 leader_election.cc:288] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ec44523932b460f973ea6c2e1fc8f74 (127.0.97.2:42831), 0ad24e75ec944c728b5bdfadf346777e (127.0.97.1:38347)
I20260924 03:14:05.558039 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:05.551562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38957 (local address 127.0.97.2:42831)
0924 03:14:05.551796 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:05.551803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:05.551823 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:14:05.551913 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:05.551922 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:05.552009 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:05.552107 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:05.552661 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.553646 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.555599 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.557021 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.557169 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:05.557277 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:05.557608 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 03:14:05.557751 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 03:14:05.558049 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:05.551046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42831 (local address 127.0.97.3:38957)
0924 03:14:05.551251 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:05.551271 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:05.551387 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:05.552232 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:05.552236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:05.552270 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:05.552511 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:05.552524 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.553775 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.553783 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:05.555410 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:05.555424 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.557199 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.557205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:05.557422 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:05.557458 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 03:14:05.557563 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 03:14:05.557725 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":3}
I20260924 03:14:05.558502 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:05.551074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37111 (local address 127.0.97.1:38347)
0924 03:14:05.551259 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:05.551267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:05.552395 (+ 1128us) server_negotiation.cc:407] Connection header received
0924 03:14:05.552544 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:05.552550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:05.552627 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:05.552722 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:05.553318 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.554493 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.556476 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.557467 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.557592 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:05.557686 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:05.558221 (+ 535us) server_negotiation.cc:299] Negotiation successful
0924 03:14:05.558336 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:14:05.559194 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:05.551060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38347 (local address 127.0.97.3:37111)
0924 03:14:05.552268 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:05.552284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:05.552400 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:05.552748 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:05.552754 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:05.552782 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:05.553113 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:05.553124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.554657 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.554663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:05.556314 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:05.556325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.557835 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.557841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:05.558011 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:05.558050 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 03:14:05.558697 (+ 647us) client_negotiation.cc:240] Negotiation successful
0924 03:14:05.558834 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"mutex_wait_us":437,"thread_start_us":427,"threads_started":1}
I20260924 03:14:05.559650 18942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270e7cc6aa764b658403f426072c6f7d" candidate_uuid: "290292b8a444429aa96648fca407cd30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec44523932b460f973ea6c2e1fc8f74" is_pre_election: true
I20260924 03:14:05.560300 18942 raft_consensus.cc:2454] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 290292b8a444429aa96648fca407cd30 in term 0.
I20260924 03:14:05.560243 18808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270e7cc6aa764b658403f426072c6f7d" candidate_uuid: "290292b8a444429aa96648fca407cd30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ad24e75ec944c728b5bdfadf346777e" is_pre_election: true
I20260924 03:14:05.560806 18808 raft_consensus.cc:2454] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 290292b8a444429aa96648fca407cd30 in term 0.
I20260924 03:14:05.561229 19011 leader_election.cc:302] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 290292b8a444429aa96648fca407cd30, 7ec44523932b460f973ea6c2e1fc8f74; no voters:
I20260924 03:14:05.561621 19146 raft_consensus.cc:2781] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:14:05.561775 19146 raft_consensus.cc:482] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:14:05.561878 19146 raft_consensus.cc:3037] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:05.563989 19146 raft_consensus.cc:504] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290292b8a444429aa96648fca407cd30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45741 } } peers { permanent_uuid: "7ec44523932b460f973ea6c2e1fc8f74" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42831 } } peers { permanent_uuid: "0ad24e75ec944c728b5bdfadf346777e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38347 } }
I20260924 03:14:05.564625 19146 leader_election.cc:288] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 [CANDIDATE]: Term 1 election: Requested vote from peers 7ec44523932b460f973ea6c2e1fc8f74 (127.0.97.2:42831), 0ad24e75ec944c728b5bdfadf346777e (127.0.97.1:38347)
I20260924 03:14:05.565001 18942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270e7cc6aa764b658403f426072c6f7d" candidate_uuid: "290292b8a444429aa96648fca407cd30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec44523932b460f973ea6c2e1fc8f74"
I20260924 03:14:05.565227 18942 raft_consensus.cc:3037] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:05.565263 18808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270e7cc6aa764b658403f426072c6f7d" candidate_uuid: "290292b8a444429aa96648fca407cd30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ad24e75ec944c728b5bdfadf346777e"
I20260924 03:14:05.565475 18808 raft_consensus.cc:3037] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:05.568595 18942 raft_consensus.cc:2454] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 290292b8a444429aa96648fca407cd30 in term 1.
I20260924 03:14:05.568631 18808 raft_consensus.cc:2454] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 290292b8a444429aa96648fca407cd30 in term 1.
I20260924 03:14:05.569059 19011 leader_election.cc:302] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 290292b8a444429aa96648fca407cd30, 7ec44523932b460f973ea6c2e1fc8f74; no voters:
I20260924 03:14:05.569355 19146 raft_consensus.cc:2781] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:05.569941 19146 raft_consensus.cc:686] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 [term 1 LEADER]: Becoming Leader. State: Replica: 290292b8a444429aa96648fca407cd30, State: Running, Role: LEADER
I20260924 03:14:05.570442 19146 consensus_queue.cc:237] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290292b8a444429aa96648fca407cd30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45741 } } peers { permanent_uuid: "7ec44523932b460f973ea6c2e1fc8f74" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42831 } } peers { permanent_uuid: "0ad24e75ec944c728b5bdfadf346777e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38347 } }
I20260924 03:14:05.575166 18663 catalog_manager.cc:5495] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 290292b8a444429aa96648fca407cd30 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "290292b8a444429aa96648fca407cd30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290292b8a444429aa96648fca407cd30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec44523932b460f973ea6c2e1fc8f74" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ad24e75ec944c728b5bdfadf346777e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38347 } health_report { overall_health: UNKNOWN } } }
I20260924 03:14:05.625921 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:05.620213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58786 (local address 127.0.97.3:45741)
0924 03:14:05.620473 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:05.620478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:05.620538 (+ 60us) server_negotiation.cc:407] Connection header received
0924 03:14:05.620723 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:05.620727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:05.620823 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:05.620951 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:14:05.621365 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.622071 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.623712 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.624093 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.624187 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:05.624741 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:14:05.625232 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:14:05.625382 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:05.625555 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 03:14:05.625626 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260924 03:14:05.634043 18808 raft_consensus.cc:1261] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e [term 1 FOLLOWER]: Refusing update from remote peer 290292b8a444429aa96648fca407cd30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:14:05.634076 18942 raft_consensus.cc:1261] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74 [term 1 FOLLOWER]: Refusing update from remote peer 290292b8a444429aa96648fca407cd30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:14:05.634961 19150 consensus_queue.cc:1035] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec44523932b460f973ea6c2e1fc8f74" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:14:05.635315 19146 consensus_queue.cc:1035] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ad24e75ec944c728b5bdfadf346777e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:14:05.646476 19153 mvcc.cc:205] Tried to move back new op lower bound from 7332739668507156480 to 7332739668260958208. Current Snapshot: MvccSnapshot[applied={T|T < 7332739668507156480}]
I20260924 03:14:05.646785 19154 mvcc.cc:205] Tried to move back new op lower bound from 7332739668507156480 to 7332739668260958208. Current Snapshot: MvccSnapshot[applied={T|T < 7332739668507156480}]
I20260924 03:14:05.663520 18664 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:14:05.663832 18664 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:14:05.667805 18664 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3fd6386a88b94eecb098601594855435: Sending DeleteTablet for 3 replicas of tablet 270e7cc6aa764b658403f426072c6f7d
I20260924 03:14:05.669351 19056 tablet_service.cc:1514] Processing DeleteTablet for tablet 270e7cc6aa764b658403f426072c6f7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:14:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58770
I20260924 03:14:05.670120 18922 tablet_service.cc:1514] Processing DeleteTablet for tablet 270e7cc6aa764b658403f426072c6f7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:14:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49380
I20260924 03:14:05.670331 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 18725
I20260924 03:14:05.671180 18788 tablet_service.cc:1514] Processing DeleteTablet for tablet 270e7cc6aa764b658403f426072c6f7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:14:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48420
I20260924 03:14:05.682523 19165 tablet_replica.cc:330] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e: stopping tablet replica
I20260924 03:14:05.683050 19165 raft_consensus.cc:2229] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:14:05.683610 19165 raft_consensus.cc:2258] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:14:05.684438 19164 tablet_replica.cc:330] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74: stopping tablet replica
I20260924 03:14:05.684857 19164 raft_consensus.cc:2229] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:14:05.685333 19164 raft_consensus.cc:2258] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:14:05.686077 19163 tablet_replica.cc:330] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30: stopping tablet replica
I20260924 03:14:05.686479 19163 raft_consensus.cc:2229] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:14:05.687182 19163 raft_consensus.cc:2258] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:14:05.689061 19165 ts_tablet_manager.cc:1905] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:14:05.689424 19163 ts_tablet_manager.cc:1905] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 03:14:05 dist-test-slave-5l98 krb5kdc[18619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:14:05.692577 19164 ts_tablet_manager.cc:1905] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:14:05.696712 19165 ts_tablet_manager.cc:1918] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:14:05.696969 19165 log.cc:1196] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/270e7cc6aa764b658403f426072c6f7d
I20260924 03:14:05.697208 19163 ts_tablet_manager.cc:1918] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:14:05.697451 19163 log.cc:1196] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/270e7cc6aa764b658403f426072c6f7d
I20260924 03:14:05.697599 19165 ts_tablet_manager.cc:1939] T 270e7cc6aa764b658403f426072c6f7d P 0ad24e75ec944c728b5bdfadf346777e: Deleting consensus metadata
I20260924 03:14:05.697993 19163 ts_tablet_manager.cc:1939] T 270e7cc6aa764b658403f426072c6f7d P 290292b8a444429aa96648fca407cd30: Deleting consensus metadata
I20260924 03:14:05.699793 18649 catalog_manager.cc:4841] TS 290292b8a444429aa96648fca407cd30 (127.0.97.3:45741): tablet 270e7cc6aa764b658403f426072c6f7d (table test-table [id=69a89715991847099399a039d0aef5b6]) successfully deleted
I20260924 03:14:05.700107 19164 ts_tablet_manager.cc:1918] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:14:05.700155 18649 catalog_manager.cc:4841] TS 0ad24e75ec944c728b5bdfadf346777e (127.0.97.1:38347): tablet 270e7cc6aa764b658403f426072c6f7d (table test-table [id=69a89715991847099399a039d0aef5b6]) successfully deleted
I20260924 03:14:05.700268 19164 log.cc:1196] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/270e7cc6aa764b658403f426072c6f7d
I20260924 03:14:05.700807 19164 ts_tablet_manager.cc:1939] T 270e7cc6aa764b658403f426072c6f7d P 7ec44523932b460f973ea6c2e1fc8f74: Deleting consensus metadata
I20260924 03:14:05.701360 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:05.672731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48432 (local address 127.0.97.1:38347)
0924 03:14:05.672906 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:05.672913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:05.672933 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:14:05.672992 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:05.672997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:05.673075 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:05.673179 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:05.682008 (+ 8829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.683230 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.685239 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.685901 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.686006 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:05.691936 (+ 5930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:05.691950 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:05.691967 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:05.692010 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:05.696216 (+ 4206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:05.697201 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:05.697232 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:05.697239 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:05.697424 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:05.698017 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:05.698022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:05.698025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:05.698527 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:05.698855 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:05.700682 (+ 1827us) server_negotiation.cc:299] Negotiation successful
0924 03:14:05.700937 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 03:14:05.702033 18848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:05.702450 18650 catalog_manager.cc:4841] TS 7ec44523932b460f973ea6c2e1fc8f74 (127.0.97.2:42831): tablet 270e7cc6aa764b658403f426072c6f7d (table test-table [id=69a89715991847099399a039d0aef5b6]) successfully deleted
I20260924 03:14:05.802939 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 18725
I20260924 03:14:05.820981 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 18859
Sep 24 03:14:05 dist-test-slave-5l98 krb5kdc[18619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 03:14:05.837036 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:05.822037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.0.97.2:42831)
0924 03:14:05.822263 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:05.822268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:05.822283 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:14:05.822338 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:05.822341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:05.822405 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:05.822519 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:05.823731 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.824543 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.825954 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.826393 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.826533 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:05.830939 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:05.830955 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:05.830972 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:05.831008 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:05.834086 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:05.834868 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:05.834876 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:05.834878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:05.834996 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:05.835385 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:05.835393 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:05.835400 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:05.835761 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:05.836067 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:05.836344 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 03:14:05.836595 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 03:14:05.837891 18982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:05.975805 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 18859
I20260924 03:14:05.993053 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 18993
Sep 24 03:14:06 dist-test-slave-5l98 krb5kdc[18619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 03:14:06.013759 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:05.993969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58794 (local address 127.0.97.3:45741)
0924 03:14:05.994204 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:05.994209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:05.994352 (+ 143us) server_negotiation.cc:407] Connection header received
0924 03:14:05.994593 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:05.994597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:05.994675 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:05.994784 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:05.996621 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.997557 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:05.999444 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:05.999906 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:06.000006 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:06.005967 (+ 5961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:06.005986 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:06.006007 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:06.006054 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:06.010549 (+ 4495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:06.011329 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:06.011338 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:06.011342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:06.011544 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:06.011945 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:06.011950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:06.011954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:06.012426 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:06.012762 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:06.013065 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 03:14:06.013326 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 03:14:06.014612 19116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:06.139840 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 18993
I20260924 03:14:06.159277 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 18632
I20260924 03:14:06.160465 18694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:06.288694 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 18632
2026-09-24T03:14:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4590 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T03:14:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:06Z Disabled control of system clock
I20260924 03:14:06.369700 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39269
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:14:06.701594 19174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:06.701892 19174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:06.701975 19174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:06.709414 19174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:06.709581 19174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:06.709697 19174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:06.709822 19174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:06.709900 19174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:06.719290 19174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39269
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:06.720716 19174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:06.722437 19174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:06.734761 19182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:06.734745 19179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:06.734719 19180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:06.735575 19174 server_base.cc:1034] running on GCE node
I20260924 03:14:06.736402 19174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:06.737926 19174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:06.739158 19174 hybrid_clock.cc:648] HybridClock initialized: now 1790219646739097 us; error 56 us; skew 500 ppm
I20260924 03:14:06.739544 19174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:14:06.742019 19174 webserver.cc:460] Webserver started at http://127.0.97.62:35907/ using document root <none> and password file <none>
I20260924 03:14:06.742776 19174 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:06.742892 19174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:06.743230 19174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:06.746284 19174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "634a5e8c19534ff99dedeb9eb8dddb40"
format_stamp: "Formatted at 2026-09-24 03:14:06 on dist-test-slave-5l98"
server_key: "202ea7a3b8e1cb63a411c03eeeeb0669"
server_key_iv: "b58a4b38e0db5b3bc773375df64290cf"
server_key_version: "encryptionkey@0"
I20260924 03:14:06.747102 19174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "634a5e8c19534ff99dedeb9eb8dddb40"
format_stamp: "Formatted at 2026-09-24 03:14:06 on dist-test-slave-5l98"
server_key: "202ea7a3b8e1cb63a411c03eeeeb0669"
server_key_iv: "b58a4b38e0db5b3bc773375df64290cf"
server_key_version: "encryptionkey@0"
I20260924 03:14:06.752437 19174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:14:06.755964 19188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:06.757344 19174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:14:06.757539 19174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "634a5e8c19534ff99dedeb9eb8dddb40"
format_stamp: "Formatted at 2026-09-24 03:14:06 on dist-test-slave-5l98"
server_key: "202ea7a3b8e1cb63a411c03eeeeb0669"
server_key_iv: "b58a4b38e0db5b3bc773375df64290cf"
server_key_version: "encryptionkey@0"
I20260924 03:14:06.757758 19174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:06.781798 19174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:06.782691 19174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:06.782990 19174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:06.802392 19174 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39269
I20260924 03:14:06.802408 19239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39269 every 8 connection(s)
I20260924 03:14:06.803788 19174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:06.804141 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 19174
I20260924 03:14:06.804317 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:06.805048 388 external_mini_cluster.cc:1428] Setting key 0a048d8992cbe1498e3bea14c4c12c43
I20260924 03:14:06.808137 19240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:06.818356 19243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:06.806970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.0.97.62:39269)
0924 03:14:06.807942 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:06.807952 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:06.807991 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:14:06.809767 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:06.809790 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:06.810215 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:06.810639 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:06.811181 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:06.812541 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:06.813945 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:06.814602 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:06.814807 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:06.816039 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:06.816049 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:06.816062 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:06.816713 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:06.816791 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:06.816812 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:06.816951 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:06.817164 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 03:14:06.817465 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":366,"threads_started":1}
I20260924 03:14:06.818735 19240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40: Bootstrap starting.
I20260924 03:14:06.822199 19240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:06.823144 19240 log.cc:824] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:06.826047 19240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40: No bootstrap required, opened a new log
I20260924 03:14:06.833647 19240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634a5e8c19534ff99dedeb9eb8dddb40" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39269 } }
I20260924 03:14:06.834040 19240 raft_consensus.cc:374] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:06.834112 19240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634a5e8c19534ff99dedeb9eb8dddb40, State: Initialized, Role: FOLLOWER
I20260924 03:14:06.834736 19240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634a5e8c19534ff99dedeb9eb8dddb40" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39269 } }
I20260924 03:14:06.835080 19240 raft_consensus.cc:388] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:06.835317 19240 raft_consensus.cc:482] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:06.835593 19240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:06.837927 19240 raft_consensus.cc:504] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634a5e8c19534ff99dedeb9eb8dddb40" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39269 } }
I20260924 03:14:06.838467 19240 leader_election.cc:302] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 634a5e8c19534ff99dedeb9eb8dddb40; no voters:
I20260924 03:14:06.839076 19240 leader_election.cc:288] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:06.839301 19245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:06.840114 19245 raft_consensus.cc:686] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40 [term 1 LEADER]: Becoming Leader. State: Replica: 634a5e8c19534ff99dedeb9eb8dddb40, State: Running, Role: LEADER
I20260924 03:14:06.840777 19245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634a5e8c19534ff99dedeb9eb8dddb40" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39269 } }
I20260924 03:14:06.841356 19240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:06.843820 19246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "634a5e8c19534ff99dedeb9eb8dddb40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634a5e8c19534ff99dedeb9eb8dddb40" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39269 } } }
I20260924 03:14:06.844314 19246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:06.844655 19247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 634a5e8c19534ff99dedeb9eb8dddb40. Latest consensus state: current_term: 1 leader_uuid: "634a5e8c19534ff99dedeb9eb8dddb40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634a5e8c19534ff99dedeb9eb8dddb40" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39269 } } }
I20260924 03:14:06.845135 19247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:06.846815 19251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:06.852005 19251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:06.863020 19251 catalog_manager.cc:1347] Generated new cluster ID: 77716ceead654d06857518e14565497b
I20260924 03:14:06.863217 19251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:06.874964 19251 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:06.876340 19251 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:06.895437 19251 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40: Generated new TSK 0
I20260924 03:14:06.896919 19251 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:14:06.907325 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39269
--builtin_ntp_servers=127.0.97.20:36973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:14:07.273432 19264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:07.273720 19264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:07.273851 19264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:07.281497 19264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:07.281608 19264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:07.281769 19264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:14:07.291546 19264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:07.292965 19264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:07.294670 19264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:07.307507 19269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:07.307462 19270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:07.307526 19272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:07.308661 19264 server_base.cc:1034] running on GCE node
I20260924 03:14:07.309237 19264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:07.310189 19264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:07.311492 19264 hybrid_clock.cc:648] HybridClock initialized: now 1790219647311424 us; error 57 us; skew 500 ppm
I20260924 03:14:07.311827 19264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:14:07.314246 19264 webserver.cc:460] Webserver started at http://127.0.97.1:44597/ using document root <none> and password file <none>
I20260924 03:14:07.314924 19264 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:07.315057 19264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:07.315369 19264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:07.318279 19264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "0c67824584f34275b09a8304a824409a"
format_stamp: "Formatted at 2026-09-24 03:14:07 on dist-test-slave-5l98"
server_key: "40c99e9c2153e707a81de24c39d4fd53"
server_key_iv: "f3f6d540c62c08ed0528f3a21c94b42b"
server_key_version: "encryptionkey@0"
I20260924 03:14:07.319149 19264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0c67824584f34275b09a8304a824409a"
format_stamp: "Formatted at 2026-09-24 03:14:07 on dist-test-slave-5l98"
server_key: "40c99e9c2153e707a81de24c39d4fd53"
server_key_iv: "f3f6d540c62c08ed0528f3a21c94b42b"
server_key_version: "encryptionkey@0"
I20260924 03:14:07.324183 19264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:14:07.327786 19278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:07.329263 19264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:14:07.329442 19264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "0c67824584f34275b09a8304a824409a"
format_stamp: "Formatted at 2026-09-24 03:14:07 on dist-test-slave-5l98"
server_key: "40c99e9c2153e707a81de24c39d4fd53"
server_key_iv: "f3f6d540c62c08ed0528f3a21c94b42b"
server_key_version: "encryptionkey@0"
I20260924 03:14:07.329701 19264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:07.360110 19264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:07.361019 19264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:07.361313 19264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:07.362454 19264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:14:07.364022 19264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:14:07.364110 19264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:07.364245 19264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:14:07.364324 19264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:07.400036 19264 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44389
I20260924 03:14:07.400158 19390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44389 every 8 connection(s)
I20260924 03:14:07.401459 19264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:14:07.408354 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 19264
I20260924 03:14:07.408622 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:14:07.409189 388 external_mini_cluster.cc:1428] Setting key 6ae3b4b60b79cd2d8237c86613fed779
I20260924 03:14:07.412060 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39269
--builtin_ntp_servers=127.0.97.20:36973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:14:07.415431 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:07.404028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36097 (local address 127.0.97.62:39269)
0924 03:14:07.404474 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:07.404479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:07.405853 (+ 1374us) server_negotiation.cc:407] Connection header received
0924 03:14:07.407693 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:07.407698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:07.407798 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:07.407943 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:07.409133 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:07.409981 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:07.412025 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:07.412775 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:07.412942 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:07.414386 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:07.414396 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:07.414403 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:07.414428 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:07.414491 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:07.414509 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:07.414602 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:07.414871 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 03:14:07.414985 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":234,"threads_started":1}
I20260924 03:14:07.416369 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:07.404719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39269 (local address 127.0.97.1:36097)
0924 03:14:07.405625 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:07.405660 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:07.407423 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:07.408154 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:07.408162 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:07.408254 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:07.408962 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:07.408985 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:07.410175 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:07.410181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:07.411834 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:07.411847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:07.413050 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:07.413055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:07.413265 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:07.414009 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:14:07.414011 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:07.414065 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:14:07.414148 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:07.414636 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:07.414646 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:07.414653 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 03:14:07.414825 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 03:14:07.415143 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":302,"threads_started":1}
I20260924 03:14:07.418087 19391 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39269
I20260924 03:14:07.418457 19391 heartbeater.cc:464] Registering TS with master...
I20260924 03:14:07.419377 19391 heartbeater.cc:511] Master 127.0.97.62:39269 requested a full tablet report, sending...
I20260924 03:14:07.422042 19205 ts_manager.cc:194] Registered new tserver with Master: 0c67824584f34275b09a8304a824409a (127.0.97.1:44389)
I20260924 03:14:07.423540 19205 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:36097
W20260924 03:14:07.745707 19395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:07.746037 19395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:07.746192 19395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:07.753758 19395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:07.753917 19395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:07.754101 19395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:14:07.763671 19395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:07.765046 19395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:07.766752 19395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:07.779906 19400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:07.779994 19403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:07.779948 19401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:07.781374 19395 server_base.cc:1034] running on GCE node
I20260924 03:14:07.782060 19395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:07.782976 19395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:07.784188 19395 hybrid_clock.cc:648] HybridClock initialized: now 1790219647784148 us; error 35 us; skew 500 ppm
I20260924 03:14:07.784463 19395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:14:07.786981 19395 webserver.cc:460] Webserver started at http://127.0.97.2:43431/ using document root <none> and password file <none>
I20260924 03:14:07.787756 19395 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:07.787845 19395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:07.788117 19395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:07.791062 19395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "d8390d3756d64895965795de1b661dbf"
format_stamp: "Formatted at 2026-09-24 03:14:07 on dist-test-slave-5l98"
server_key: "1edab651038e8ad60c8e69335204da16"
server_key_iv: "0a2921a038e6b6cedfabdf7208e4812b"
server_key_version: "encryptionkey@0"
I20260924 03:14:07.791838 19395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d8390d3756d64895965795de1b661dbf"
format_stamp: "Formatted at 2026-09-24 03:14:07 on dist-test-slave-5l98"
server_key: "1edab651038e8ad60c8e69335204da16"
server_key_iv: "0a2921a038e6b6cedfabdf7208e4812b"
server_key_version: "encryptionkey@0"
I20260924 03:14:07.797564 19395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 03:14:07.801252 19409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:07.802733 19395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:14:07.802927 19395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "d8390d3756d64895965795de1b661dbf"
format_stamp: "Formatted at 2026-09-24 03:14:07 on dist-test-slave-5l98"
server_key: "1edab651038e8ad60c8e69335204da16"
server_key_iv: "0a2921a038e6b6cedfabdf7208e4812b"
server_key_version: "encryptionkey@0"
I20260924 03:14:07.803153 19395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:07.821183 19395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:07.822120 19395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:07.822453 19395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:07.823514 19395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:14:07.825227 19395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:14:07.825368 19395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:07.825534 19395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:14:07.825634 19395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:07.861941 19395 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39913
I20260924 03:14:07.861989 19521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39913 every 8 connection(s)
I20260924 03:14:07.863332 19395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:14:07.870147 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 19395
I20260924 03:14:07.870352 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:14:07.870823 388 external_mini_cluster.cc:1428] Setting key 34f09c7b29a4a0fc26a44319782ef03c
I20260924 03:14:07.873889 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39269
--builtin_ntp_servers=127.0.97.20:36973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:14:07.877915 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:07.866150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44717 (local address 127.0.97.62:39269)
0924 03:14:07.866368 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:07.866373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:07.867838 (+ 1465us) server_negotiation.cc:407] Connection header received
0924 03:14:07.869791 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:07.869799 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:07.869911 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:07.870044 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:07.871387 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:07.872336 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:07.874304 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:07.875149 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:07.875301 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:07.876871 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:07.876880 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:07.876887 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:07.876914 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:07.876958 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:07.876977 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:07.877075 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:07.877365 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 03:14:07.877482 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":95}
I20260924 03:14:07.878793 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:07.866699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39269 (local address 127.0.97.2:44717)
0924 03:14:07.867594 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:07.867630 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:07.869443 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:07.870272 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:07.870281 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:07.870429 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:07.871196 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:07.871215 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:07.872525 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:07.872530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:07.874156 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:07.874168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:07.875392 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:07.875398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:07.875609 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:07.876457 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:14:07.876459 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:07.876554 (+ 95us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:14:07.876641 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:07.877109 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:07.877118 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:07.877124 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 03:14:07.877301 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 03:14:07.877622 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":294,"threads_started":1}
I20260924 03:14:07.880441 19522 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39269
I20260924 03:14:07.880805 19522 heartbeater.cc:464] Registering TS with master...
I20260924 03:14:07.881646 19522 heartbeater.cc:511] Master 127.0.97.62:39269 requested a full tablet report, sending...
I20260924 03:14:07.883541 19205 ts_manager.cc:194] Registered new tserver with Master: d8390d3756d64895965795de1b661dbf (127.0.97.2:39913)
I20260924 03:14:07.884416 19205 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:44717
W20260924 03:14:08.229916 19525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:08.230243 19525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:08.230403 19525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:08.238197 19525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:08.238339 19525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:08.238524 19525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:14:08.247876 19525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:08.249358 19525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:08.251161 19525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:08.265538 19530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:08.265578 19531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:08.265502 19533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:08.266501 19525 server_base.cc:1034] running on GCE node
I20260924 03:14:08.267307 19525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:08.268381 19525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:08.269650 19525 hybrid_clock.cc:648] HybridClock initialized: now 1790219648269572 us; error 59 us; skew 500 ppm
I20260924 03:14:08.270102 19525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:14:08.272715 19525 webserver.cc:460] Webserver started at http://127.0.97.3:33705/ using document root <none> and password file <none>
I20260924 03:14:08.273494 19525 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:08.273656 19525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:08.274019 19525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:08.277231 19525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "744d63d65a8e47e69c95f9d8148599a2"
format_stamp: "Formatted at 2026-09-24 03:14:08 on dist-test-slave-5l98"
server_key: "abf19f65ec58b138d673da42f5118e19"
server_key_iv: "a50a3aeb3809ca3ffff662aeaf9eb654"
server_key_version: "encryptionkey@0"
I20260924 03:14:08.278118 19525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "744d63d65a8e47e69c95f9d8148599a2"
format_stamp: "Formatted at 2026-09-24 03:14:08 on dist-test-slave-5l98"
server_key: "abf19f65ec58b138d673da42f5118e19"
server_key_iv: "a50a3aeb3809ca3ffff662aeaf9eb654"
server_key_version: "encryptionkey@0"
I20260924 03:14:08.283833 19525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 03:14:08.287672 19539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:08.289192 19525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:14:08.289378 19525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "744d63d65a8e47e69c95f9d8148599a2"
format_stamp: "Formatted at 2026-09-24 03:14:08 on dist-test-slave-5l98"
server_key: "abf19f65ec58b138d673da42f5118e19"
server_key_iv: "a50a3aeb3809ca3ffff662aeaf9eb654"
server_key_version: "encryptionkey@0"
I20260924 03:14:08.289613 19525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:08.310087 19525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:08.310964 19525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:08.311282 19525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:08.312266 19525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:14:08.313805 19525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:14:08.313926 19525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:08.314064 19525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:14:08.314154 19525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:08.349205 19525 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40697
I20260924 03:14:08.349359 19651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40697 every 8 connection(s)
I20260924 03:14:08.350509 19525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:14:08.350791 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 19525
I20260924 03:14:08.350984 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:14:08.351666 388 external_mini_cluster.cc:1428] Setting key 81dbb54fc6729b12fc59f068df3ba433
I20260924 03:14:08.364095 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:08.353594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51297 (local address 127.0.97.62:39269)
0924 03:14:08.353780 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:08.353786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:08.355612 (+ 1826us) server_negotiation.cc:407] Connection header received
0924 03:14:08.357522 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:08.357527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:08.357615 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:08.357742 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:08.358769 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.359514 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.361221 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.361628 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.361785 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:08.363266 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:08.363272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:08.363277 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:08.363307 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:08.363337 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:08.363348 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:08.363443 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:08.363660 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 03:14:08.363760 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 03:14:08.365258 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:08.354313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39269 (local address 127.0.97.3:51297)
0924 03:14:08.355458 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:08.355495 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:08.357256 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:08.357960 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:08.357968 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:08.358054 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:08.358609 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:08.358625 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.359683 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.359686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:08.361076 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:08.361088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.361893 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.361902 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:08.362209 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:08.362939 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:14:08.362941 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:08.362989 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:14:08.363072 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:08.363441 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:08.363453 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:08.363463 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 03:14:08.363650 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 03:14:08.364040 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":381,"threads_started":1}
I20260924 03:14:08.367281 19652 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39269
I20260924 03:14:08.367624 19652 heartbeater.cc:464] Registering TS with master...
I20260924 03:14:08.368413 19652 heartbeater.cc:511] Master 127.0.97.62:39269 requested a full tablet report, sending...
I20260924 03:14:08.369906 19205 ts_manager.cc:194] Registered new tserver with Master: 744d63d65a8e47e69c95f9d8148599a2 (127.0.97.3:40697)
I20260924 03:14:08.370666 19205 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:51297
I20260924 03:14:08.381209 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:14:08.392151 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:08.385657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.0.97.62:39269)
0924 03:14:08.386016 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:08.386023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:08.386308 (+ 285us) server_negotiation.cc:407] Connection header received
0924 03:14:08.386536 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:08.386541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:08.386617 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:08.386725 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:08.387224 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.388017 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.389537 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.389992 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.390128 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:08.391387 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:08.391392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:08.391395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:08.391415 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:08.391446 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:08.391457 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:08.391609 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:08.391800 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 03:14:08.391870 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260924 03:14:08.396577 19205 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:14:08.405385 19205 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:14:08.427373 19391 heartbeater.cc:503] Master 127.0.97.62:39269 was elected leader, sending a full tablet report...
I20260924 03:14:08.443018 19664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:08.418825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40697 (local address 127.0.0.1:44342)
0924 03:14:08.432847 (+ 14022us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:08.432864 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:08.433078 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:08.434239 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:08.434246 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:08.434329 (+ 83us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:08.434765 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:08.434780 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.436868 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.436880 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:08.439499 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:08.439515 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.441898 (+ 2383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.441905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:08.442124 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:08.442187 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 03:14:08.442340 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 03:14:08.442524 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13819,"thread_start_us":10930,"threads_started":1}
I20260924 03:14:08.443723 19667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:08.418901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44342 (local address 127.0.97.3:40697)
0924 03:14:08.431262 (+ 12361us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:08.431275 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:08.432968 (+ 1693us) server_negotiation.cc:407] Connection header received
0924 03:14:08.433342 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:08.433350 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:08.433642 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:08.433824 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:08.435021 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.436694 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.439720 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.441339 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.441463 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:08.441748 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:08.443200 (+ 1452us) server_negotiation.cc:299] Negotiation successful
0924 03:14:08.443365 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12138,"thread_start_us":12385,"threads_started":1}
I20260924 03:14:08.443786 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:08.418997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44389 (local address 127.0.0.1:33600)
0924 03:14:08.435896 (+ 16899us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:08.435913 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:08.436023 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:08.436696 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:08.436702 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:08.436726 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:08.437252 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:08.437263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.438927 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.438932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:08.440983 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:08.440999 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.443034 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.443041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:08.443216 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:08.443249 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 03:14:08.443331 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 03:14:08.443447 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16764,"thread_start_us":16635,"threads_started":1}
I20260924 03:14:08.444391 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:08.419833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33600 (local address 127.0.97.1:44389)
0924 03:14:08.435492 (+ 15659us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:08.435504 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:08.436147 (+ 643us) server_negotiation.cc:407] Connection header received
0924 03:14:08.436237 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:08.436244 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:08.436476 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:08.436687 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:08.437469 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.438717 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.441268 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.442692 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.442821 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:08.443079 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:08.443836 (+ 757us) server_negotiation.cc:299] Negotiation successful
0924 03:14:08.444013 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15472,"thread_start_us":15329,"threads_started":1}
I20260924 03:14:08.444411 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:08.418360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39913 (local address 127.0.0.1:48820)
0924 03:14:08.430208 (+ 11848us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:08.430252 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:08.430463 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:08.437360 (+ 6897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:08.437364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:08.437393 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:08.437865 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:08.437879 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.439880 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.439886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:08.441747 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:08.441761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.443707 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.443714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:08.443924 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:08.443959 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 03:14:08.444063 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 03:14:08.444183 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11575,"mutex_wait_us":36,"thread_start_us":14148,"threads_started":1}
I20260924 03:14:08.445338 19665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:08.418546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48820 (local address 127.0.97.2:39913)
0924 03:14:08.436304 (+ 17758us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:08.436310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:08.436336 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:14:08.436425 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:08.436430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:08.436763 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:08.436971 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:08.438084 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.439746 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.441949 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.443227 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.443366 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:08.443568 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:08.444794 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0924 03:14:08.444983 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17606,"thread_start_us":17491,"threads_started":1}
I20260924 03:14:08.447924 19587 tablet_service.cc:1467] Processing CreateTablet for tablet b7d27f7af442453ebfcaf036e76d427d (DEFAULT_TABLE table=test-table [id=0842646d349b4b8cb8c354e816d0d3c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:14:08.448112 19326 tablet_service.cc:1467] Processing CreateTablet for tablet b7d27f7af442453ebfcaf036e76d427d (DEFAULT_TABLE table=test-table [id=0842646d349b4b8cb8c354e816d0d3c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:14:08.449328 19587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7d27f7af442453ebfcaf036e76d427d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:08.449509 19326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7d27f7af442453ebfcaf036e76d427d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:08.449199 19457 tablet_service.cc:1467] Processing CreateTablet for tablet b7d27f7af442453ebfcaf036e76d427d (DEFAULT_TABLE table=test-table [id=0842646d349b4b8cb8c354e816d0d3c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:14:08.450497 19457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7d27f7af442453ebfcaf036e76d427d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:08.461207 19670 tablet_bootstrap.cc:492] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2: Bootstrap starting.
I20260924 03:14:08.464228 19671 tablet_bootstrap.cc:492] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a: Bootstrap starting.
I20260924 03:14:08.466015 19672 tablet_bootstrap.cc:492] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf: Bootstrap starting.
I20260924 03:14:08.466383 19670 tablet_bootstrap.cc:654] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:08.467729 19670 log.cc:824] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:08.467978 19671 tablet_bootstrap.cc:654] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:08.469044 19671 log.cc:824] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:08.469426 19672 tablet_bootstrap.cc:654] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:08.470650 19672 log.cc:824] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:08.471207 19670 tablet_bootstrap.cc:492] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2: No bootstrap required, opened a new log
I20260924 03:14:08.471519 19670 ts_tablet_manager.cc:1397] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260924 03:14:08.471654 19671 tablet_bootstrap.cc:492] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a: No bootstrap required, opened a new log
I20260924 03:14:08.471940 19671 ts_tablet_manager.cc:1397] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 03:14:08.473620 19672 tablet_bootstrap.cc:492] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf: No bootstrap required, opened a new log
I20260924 03:14:08.473929 19672 ts_tablet_manager.cc:1397] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 03:14:08.479387 19671 raft_consensus.cc:348] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8390d3756d64895965795de1b661dbf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39913 } } peers { permanent_uuid: "744d63d65a8e47e69c95f9d8148599a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40697 } } peers { permanent_uuid: "0c67824584f34275b09a8304a824409a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 } }
I20260924 03:14:08.479787 19671 raft_consensus.cc:374] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:08.479897 19671 raft_consensus.cc:729] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c67824584f34275b09a8304a824409a, State: Initialized, Role: FOLLOWER
I20260924 03:14:08.480580 19671 consensus_queue.cc:260] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8390d3756d64895965795de1b661dbf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39913 } } peers { permanent_uuid: "744d63d65a8e47e69c95f9d8148599a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40697 } } peers { permanent_uuid: "0c67824584f34275b09a8304a824409a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 } }
I20260924 03:14:08.482398 19671 ts_tablet_manager.cc:1428] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260924 03:14:08.482647 19670 raft_consensus.cc:348] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8390d3756d64895965795de1b661dbf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39913 } } peers { permanent_uuid: "744d63d65a8e47e69c95f9d8148599a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40697 } } peers { permanent_uuid: "0c67824584f34275b09a8304a824409a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 } }
I20260924 03:14:08.483125 19670 raft_consensus.cc:374] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:08.483237 19670 raft_consensus.cc:729] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744d63d65a8e47e69c95f9d8148599a2, State: Initialized, Role: FOLLOWER
I20260924 03:14:08.483848 19670 consensus_queue.cc:260] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8390d3756d64895965795de1b661dbf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39913 } } peers { permanent_uuid: "744d63d65a8e47e69c95f9d8148599a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40697 } } peers { permanent_uuid: "0c67824584f34275b09a8304a824409a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 } }
I20260924 03:14:08.485045 19652 heartbeater.cc:503] Master 127.0.97.62:39269 was elected leader, sending a full tablet report...
I20260924 03:14:08.485709 19670 ts_tablet_manager.cc:1428] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260924 03:14:08.486402 19672 raft_consensus.cc:348] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8390d3756d64895965795de1b661dbf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39913 } } peers { permanent_uuid: "744d63d65a8e47e69c95f9d8148599a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40697 } } peers { permanent_uuid: "0c67824584f34275b09a8304a824409a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 } }
I20260924 03:14:08.486929 19672 raft_consensus.cc:374] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:08.487046 19672 raft_consensus.cc:729] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8390d3756d64895965795de1b661dbf, State: Initialized, Role: FOLLOWER
I20260924 03:14:08.487717 19672 consensus_queue.cc:260] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8390d3756d64895965795de1b661dbf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39913 } } peers { permanent_uuid: "744d63d65a8e47e69c95f9d8148599a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40697 } } peers { permanent_uuid: "0c67824584f34275b09a8304a824409a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 } }
I20260924 03:14:08.488886 19522 heartbeater.cc:503] Master 127.0.97.62:39269 was elected leader, sending a full tablet report...
I20260924 03:14:08.489516 19672 ts_tablet_manager.cc:1428] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260924 03:14:08.582827 19678 raft_consensus.cc:482] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:14:08.583395 19678 raft_consensus.cc:504] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8390d3756d64895965795de1b661dbf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39913 } } peers { permanent_uuid: "744d63d65a8e47e69c95f9d8148599a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40697 } } peers { permanent_uuid: "0c67824584f34275b09a8304a824409a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 } }
I20260924 03:14:08.585457 19678 leader_election.cc:288] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744d63d65a8e47e69c95f9d8148599a2 (127.0.97.3:40697), 0c67824584f34275b09a8304a824409a (127.0.97.1:44389)
I20260924 03:14:08.594744 19667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:08.586015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58295 (local address 127.0.97.3:40697)
0924 03:14:08.586228 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:08.586237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:08.586819 (+ 582us) server_negotiation.cc:407] Connection header received
0924 03:14:08.587047 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:08.587052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:08.587122 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:08.587223 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:08.588094 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.589221 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.592206 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.593542 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.593660 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:08.593782 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:08.594342 (+ 560us) server_negotiation.cc:299] Negotiation successful
0924 03:14:08.594534 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 03:14:08.594781 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:08.586007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59739 (local address 127.0.97.1:44389)
0924 03:14:08.586247 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:08.586257 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:08.586806 (+ 549us) server_negotiation.cc:407] Connection header received
0924 03:14:08.587010 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:08.587016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:08.587113 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:08.587223 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:08.588086 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.589260 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.592206 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.593515 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.593660 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:08.593771 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:08.594386 (+ 615us) server_negotiation.cc:299] Negotiation successful
0924 03:14:08.594541 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 03:14:08.594863 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:08.585945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40697 (local address 127.0.97.2:58295)
0924 03:14:08.586700 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:08.586715 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:08.586836 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:08.587259 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:08.587264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:08.587293 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:08.587922 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:08.587935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.589410 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.589417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:08.591977 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:08.591994 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.593874 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.593881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:08.594109 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:08.594159 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 03:14:08.594271 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 03:14:08.594481 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":181,"thread_start_us":436,"threads_started":1}
I20260924 03:14:08.595417 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:08.585779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44389 (local address 127.0.97.2:59739)
0924 03:14:08.586638 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:08.586658 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:08.586867 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:08.587259 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:08.587264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:08.587303 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:08.587859 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:08.587880 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.589404 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.589409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:08.592018 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:08.592030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.593816 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.593823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:08.594105 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:08.594231 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 03:14:08.595031 (+ 800us) client_negotiation.cc:240] Negotiation successful
0924 03:14:08.595150 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":405,"spinlock_wait_cycles":48768,"thread_start_us":477,"threads_started":1}
I20260924 03:14:08.596316 19607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d27f7af442453ebfcaf036e76d427d" candidate_uuid: "d8390d3756d64895965795de1b661dbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744d63d65a8e47e69c95f9d8148599a2" is_pre_election: true
I20260924 03:14:08.596578 19346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d27f7af442453ebfcaf036e76d427d" candidate_uuid: "d8390d3756d64895965795de1b661dbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c67824584f34275b09a8304a824409a" is_pre_election: true
I20260924 03:14:08.596911 19607 raft_consensus.cc:2454] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8390d3756d64895965795de1b661dbf in term 0.
I20260924 03:14:08.597190 19346 raft_consensus.cc:2454] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8390d3756d64895965795de1b661dbf in term 0.
I20260924 03:14:08.597616 19410 leader_election.cc:302] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744d63d65a8e47e69c95f9d8148599a2, d8390d3756d64895965795de1b661dbf; no voters:
I20260924 03:14:08.598029 19678 raft_consensus.cc:2781] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:14:08.598181 19678 raft_consensus.cc:482] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:14:08.598328 19678 raft_consensus.cc:3037] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:08.600571 19678 raft_consensus.cc:504] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8390d3756d64895965795de1b661dbf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39913 } } peers { permanent_uuid: "744d63d65a8e47e69c95f9d8148599a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40697 } } peers { permanent_uuid: "0c67824584f34275b09a8304a824409a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 } }
I20260924 03:14:08.601238 19678 leader_election.cc:288] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf [CANDIDATE]: Term 1 election: Requested vote from peers 744d63d65a8e47e69c95f9d8148599a2 (127.0.97.3:40697), 0c67824584f34275b09a8304a824409a (127.0.97.1:44389)
I20260924 03:14:08.601765 19607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d27f7af442453ebfcaf036e76d427d" candidate_uuid: "d8390d3756d64895965795de1b661dbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744d63d65a8e47e69c95f9d8148599a2"
I20260924 03:14:08.601778 19346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d27f7af442453ebfcaf036e76d427d" candidate_uuid: "d8390d3756d64895965795de1b661dbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c67824584f34275b09a8304a824409a"
I20260924 03:14:08.601989 19346 raft_consensus.cc:3037] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:08.601989 19607 raft_consensus.cc:3037] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2 [term 0 FOLLOWER]: Advancing to term 1
W20260924 03:14:08.602443 19653 tablet.cc:2369] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:14:08.605376 19607 raft_consensus.cc:2454] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8390d3756d64895965795de1b661dbf in term 1.
I20260924 03:14:08.605389 19346 raft_consensus.cc:2454] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8390d3756d64895965795de1b661dbf in term 1.
I20260924 03:14:08.605865 19410 leader_election.cc:302] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0c67824584f34275b09a8304a824409a, 744d63d65a8e47e69c95f9d8148599a2, d8390d3756d64895965795de1b661dbf; no voters:
I20260924 03:14:08.606223 19678 raft_consensus.cc:2781] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:08.606721 19678 raft_consensus.cc:686] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf [term 1 LEADER]: Becoming Leader. State: Replica: d8390d3756d64895965795de1b661dbf, State: Running, Role: LEADER
I20260924 03:14:08.607223 19678 consensus_queue.cc:237] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8390d3756d64895965795de1b661dbf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39913 } } peers { permanent_uuid: "744d63d65a8e47e69c95f9d8148599a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40697 } } peers { permanent_uuid: "0c67824584f34275b09a8304a824409a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 } }
I20260924 03:14:08.611943 19205 catalog_manager.cc:5495] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf reported cstate change: term changed from 0 to 1, leader changed from <none> to d8390d3756d64895965795de1b661dbf (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d8390d3756d64895965795de1b661dbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8390d3756d64895965795de1b661dbf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744d63d65a8e47e69c95f9d8148599a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c67824584f34275b09a8304a824409a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 } health_report { overall_health: UNKNOWN } } }
W20260924 03:14:08.615461 19523 tablet.cc:2369] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:14:08.642722 19665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:08.636585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48826 (local address 127.0.97.2:39913)
0924 03:14:08.636900 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:08.636909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:08.636945 (+ 36us) server_negotiation.cc:407] Connection header received
0924 03:14:08.637052 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:08.637056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:08.637142 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:08.637314 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:14:08.637705 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.638431 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.639924 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.640352 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.640531 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:08.641182 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:14:08.641898 (+ 716us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:14:08.642044 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:08.642326 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 03:14:08.642365 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:14:08.652101 19346 raft_consensus.cc:1261] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a [term 1 FOLLOWER]: Refusing update from remote peer d8390d3756d64895965795de1b661dbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:14:08.652089 19607 raft_consensus.cc:1261] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2 [term 1 FOLLOWER]: Refusing update from remote peer d8390d3756d64895965795de1b661dbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:14:08.653365 19678 consensus_queue.cc:1035] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c67824584f34275b09a8304a824409a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 03:14:08.653957 19392 tablet.cc:2369] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:14:08.654135 19681 consensus_queue.cc:1035] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "744d63d65a8e47e69c95f9d8148599a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:14:08.678637 19204 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:14:08.679100 19204 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:14:08.683985 19204 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 634a5e8c19534ff99dedeb9eb8dddb40: Sending DeleteTablet for 3 replicas of tablet b7d27f7af442453ebfcaf036e76d427d
I20260924 03:14:08.685508 19457 tablet_service.cc:1514] Processing DeleteTablet for tablet b7d27f7af442453ebfcaf036e76d427d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:14:08 UTC) from {username='slave'} at 127.0.0.1:48820
I20260924 03:14:08.685508 19587 tablet_service.cc:1514] Processing DeleteTablet for tablet b7d27f7af442453ebfcaf036e76d427d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:14:08 UTC) from {username='slave'} at 127.0.0.1:44342
I20260924 03:14:08.685796 19326 tablet_service.cc:1514] Processing DeleteTablet for tablet b7d27f7af442453ebfcaf036e76d427d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:14:08 UTC) from {username='slave'} at 127.0.0.1:33600
I20260924 03:14:08.686717 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 19264
I20260924 03:14:08.697969 19694 tablet_replica.cc:330] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2: stopping tablet replica
I20260924 03:14:08.698412 19694 raft_consensus.cc:2229] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:14:08.698918 19694 raft_consensus.cc:2258] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:14:08.699734 19696 tablet_replica.cc:330] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a: stopping tablet replica
I20260924 03:14:08.700098 19696 raft_consensus.cc:2229] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:14:08.700555 19696 raft_consensus.cc:2258] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:14:08.701834 19695 tablet_replica.cc:330] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf: stopping tablet replica
I20260924 03:14:08.702222 19695 raft_consensus.cc:2229] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:14:08.702749 19694 ts_tablet_manager.cc:1905] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:14:08.702852 19695 raft_consensus.cc:2258] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:14:08.702903 19696 ts_tablet_manager.cc:1905] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:14:08.710786 19694 ts_tablet_manager.cc:1918] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:14:08.710916 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:08.688172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33606 (local address 127.0.97.1:44389)
0924 03:14:08.688426 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:08.688435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:08.700574 (+ 12139us) server_negotiation.cc:407] Connection header received
0924 03:14:08.701433 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:08.701440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:08.701551 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:08.701651 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:08.702403 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.703703 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.706299 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.707113 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.707226 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:08.708947 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:08.708965 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:08.708984 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:08.709870 (+ 886us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:08.709933 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:08.709959 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:08.710056 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:08.710311 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 03:14:08.710432 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 03:14:08.711036 19696 ts_tablet_manager.cc:1918] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:14:08.711238 19696 log.cc:1196] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/b7d27f7af442453ebfcaf036e76d427d
I20260924 03:14:08.711380 19694 log.cc:1196] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/b7d27f7af442453ebfcaf036e76d427d
I20260924 03:14:08.711853 19696 ts_tablet_manager.cc:1939] T b7d27f7af442453ebfcaf036e76d427d P 0c67824584f34275b09a8304a824409a: Deleting consensus metadata
I20260924 03:14:08.712056 19694 ts_tablet_manager.cc:1939] T b7d27f7af442453ebfcaf036e76d427d P 744d63d65a8e47e69c95f9d8148599a2: Deleting consensus metadata
I20260924 03:14:08.712308 19386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:08.714084 19189 catalog_manager.cc:4841] TS 744d63d65a8e47e69c95f9d8148599a2 (127.0.97.3:40697): tablet b7d27f7af442453ebfcaf036e76d427d (table test-table [id=0842646d349b4b8cb8c354e816d0d3c5]) successfully deleted
I20260924 03:14:08.714397 19695 ts_tablet_manager.cc:1905] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:14:08.722822 19695 ts_tablet_manager.cc:1918] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:14:08.723049 19695 log.cc:1196] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/b7d27f7af442453ebfcaf036e76d427d
I20260924 03:14:08.723665 19695 ts_tablet_manager.cc:1939] T b7d27f7af442453ebfcaf036e76d427d P d8390d3756d64895965795de1b661dbf: Deleting consensus metadata
I20260924 03:14:08.725848 19190 catalog_manager.cc:4841] TS d8390d3756d64895965795de1b661dbf (127.0.97.2:39913): tablet b7d27f7af442453ebfcaf036e76d427d (table test-table [id=0842646d349b4b8cb8c354e816d0d3c5]) successfully deleted
I20260924 03:14:08.813781 19192 catalog_manager.cc:4841] TS 0c67824584f34275b09a8304a824409a (127.0.97.1:44389): tablet b7d27f7af442453ebfcaf036e76d427d (table test-table [id=0842646d349b4b8cb8c354e816d0d3c5]) successfully deleted
I20260924 03:14:08.814285 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 19264
I20260924 03:14:08.832100 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 19395
I20260924 03:14:08.840683 19665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:08.833025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48828 (local address 127.0.97.2:39913)
0924 03:14:08.833191 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:08.833195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:08.833213 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:14:08.833448 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:08.833453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:08.833553 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:08.833645 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:08.834119 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.834939 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.836943 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.837399 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.837510 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:08.838874 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:08.838885 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:08.838900 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:08.839556 (+ 656us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:08.839609 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:08.839631 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:08.839720 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:08.840192 (+ 472us) server_negotiation.cc:299] Negotiation successful
0924 03:14:08.840334 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 03:14:08.841562 19517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:08.956718 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 19395
I20260924 03:14:08.975040 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 19525
I20260924 03:14:08.982301 19667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:08.975810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44348 (local address 127.0.97.3:40697)
0924 03:14:08.976048 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:08.976054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:08.976157 (+ 103us) server_negotiation.cc:407] Connection header received
0924 03:14:08.976372 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:08.976393 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:08.976533 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:08.976652 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:08.977104 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.977842 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.979162 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:08.979557 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:08.979695 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:08.980965 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:08.980974 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:08.980988 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:08.981566 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:08.981617 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:08.981637 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:08.981764 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:08.981893 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 03:14:08.981997 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 03:14:08.982986 19647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:09.075614 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 19525
I20260924 03:14:09.092767 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 19174
I20260924 03:14:09.093914 19235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:09.206388 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 19174
2026-09-24T03:14:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2915 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:09 dist-test-slave-5l98 krb5kdc[19704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:09 dist-test-slave-5l98 krb5kdc[19704](info): set up 2 sockets
Sep 24 03:14:09 dist-test-slave-5l98 krb5kdc[19704](info): commencing operation
krb5kdc: starting...
W20260924 03:14:10.320878 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:10 dist-test-slave-5l98 krb5kdc[19704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:14:10.605957 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39447
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:10.951822 19717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:10.952138 19717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:10.952342 19717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:10.960036 19717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:10.960166 19717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:10.960264 19717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:10.960353 19717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:10.960436 19717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:10.971006 19717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:10.972438 19717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:10.973877 19717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:10.987008 19722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:10.986982 19723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:10.987435 19725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:10.987848 19717 server_base.cc:1034] running on GCE node
I20260924 03:14:10.988945 19717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:10.990625 19717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:10.992125 19717 hybrid_clock.cc:648] HybridClock initialized: now 1790219650992056 us; error 47 us; skew 500 ppm
I20260924 03:14:10.992638 19717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:10 dist-test-slave-5l98 krb5kdc[19704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:11.002617 19717 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:14:11.004325 19717 webserver.cc:460] Webserver started at http://127.0.97.62:40093/ using document root <none> and password file <none>
I20260924 03:14:11.005118 19717 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:11.005228 19717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:11.005501 19717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:11.008672 19717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "243f8160e6894cd4aca1866d944ad637"
format_stamp: "Formatted at 2026-09-24 03:14:11 on dist-test-slave-5l98"
server_key: "a5485cb8ab8f7893d14e580649550fac"
server_key_iv: "07c53eaf40b6a65a0ddad9835817cec0"
server_key_version: "encryptionkey@0"
I20260924 03:14:11.009819 19717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "243f8160e6894cd4aca1866d944ad637"
format_stamp: "Formatted at 2026-09-24 03:14:11 on dist-test-slave-5l98"
server_key: "a5485cb8ab8f7893d14e580649550fac"
server_key_iv: "07c53eaf40b6a65a0ddad9835817cec0"
server_key_version: "encryptionkey@0"
I20260924 03:14:11.015786 19717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:14:11.019771 19732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:11.021467 19717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 03:14:11.021656 19717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "243f8160e6894cd4aca1866d944ad637"
format_stamp: "Formatted at 2026-09-24 03:14:11 on dist-test-slave-5l98"
server_key: "a5485cb8ab8f7893d14e580649550fac"
server_key_iv: "07c53eaf40b6a65a0ddad9835817cec0"
server_key_version: "encryptionkey@0"
I20260924 03:14:11.021898 19717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:11.046730 19717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:11.051506 19717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:11.051884 19717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:11.072939 19717 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39447
I20260924 03:14:11.072942 19783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39447 every 8 connection(s)
I20260924 03:14:11.074551 19717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:11.078771 19784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:11.083453 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 19717
I20260924 03:14:11.083642 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:11.084245 388 external_mini_cluster.cc:1428] Setting key 8f62769281a552b9fb64722c637f2586
I20260924 03:14:11.091110 19784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637: Bootstrap starting.
I20260924 03:14:11.094309 19784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:11.095343 19784 log.cc:824] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:11.098680 19784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637: No bootstrap required, opened a new log
Sep 24 03:14:11 dist-test-slave-5l98 krb5kdc[19704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:14:11.110077 19784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243f8160e6894cd4aca1866d944ad637" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39447 } }
I20260924 03:14:11.110615 19784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:11.110741 19784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 243f8160e6894cd4aca1866d944ad637, State: Initialized, Role: FOLLOWER
I20260924 03:14:11.110891 19787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:11.086468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.0.97.62:39447)
0924 03:14:11.087285 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:11.087298 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:11.087337 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:14:11.089708 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:11.089734 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:11.090174 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:11.090574 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:11.092179 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:11.093580 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:11.095002 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:11.095767 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:11.096167 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:11.100759 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:11.100774 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:11.100802 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:11.100870 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:11.105565 (+ 4695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:11.106513 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:11.106523 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:11.106537 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:11.106788 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:11.107233 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:11.107241 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:11.107248 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:11.108079 (+ 831us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:11.108407 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:11.108887 (+ 480us) server_negotiation.cc:299] Negotiation successful
0924 03:14:11.109391 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":381,"threads_started":1}
I20260924 03:14:11.111414 19784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243f8160e6894cd4aca1866d944ad637" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39447 } }
I20260924 03:14:11.111693 19784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:11.111867 19784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:11.112066 19784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:11.115119 19784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243f8160e6894cd4aca1866d944ad637" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39447 } }
I20260924 03:14:11.115656 19784 leader_election.cc:302] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 243f8160e6894cd4aca1866d944ad637; no voters:
I20260924 03:14:11.116259 19784 leader_election.cc:288] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:11.116430 19789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:11.117280 19789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637 [term 1 LEADER]: Becoming Leader. State: Replica: 243f8160e6894cd4aca1866d944ad637, State: Running, Role: LEADER
I20260924 03:14:11.117905 19789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243f8160e6894cd4aca1866d944ad637" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39447 } }
I20260924 03:14:11.118573 19784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:11.121119 19790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "243f8160e6894cd4aca1866d944ad637" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243f8160e6894cd4aca1866d944ad637" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39447 } } }
I20260924 03:14:11.121465 19790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:11.121062 19791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 243f8160e6894cd4aca1866d944ad637. Latest consensus state: current_term: 1 leader_uuid: "243f8160e6894cd4aca1866d944ad637" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243f8160e6894cd4aca1866d944ad637" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39447 } } }
I20260924 03:14:11.121628 19791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:11.123073 19796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:11.129258 19796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:11.141583 19796 catalog_manager.cc:1347] Generated new cluster ID: 1ee3846dc19743a8a6903b0db3513d80
I20260924 03:14:11.141744 19796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:11.164099 19796 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:11.165575 19796 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:11.175377 19796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637: Generated new TSK 0
I20260924 03:14:11.176362 19796 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 03:14:11.261716 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39447
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:11.625802 19810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:11.626255 19810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:11.626446 19810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:11.634430 19810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:11.634595 19810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:11.634797 19810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:14:11.645066 19810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:11.646466 19810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:11.647902 19810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:11.662457 19818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:11.662418 19815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:11.662487 19816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:11.663719 19810 server_base.cc:1034] running on GCE node
I20260924 03:14:11.664381 19810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:11.665503 19810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:11.666761 19810 hybrid_clock.cc:648] HybridClock initialized: now 1790219651666716 us; error 58 us; skew 500 ppm
I20260924 03:14:11.667148 19810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:11 dist-test-slave-5l98 krb5kdc[19704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:11.670673 19810 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 03:14:11.672580 19810 webserver.cc:460] Webserver started at http://127.0.97.1:34961/ using document root <none> and password file <none>
I20260924 03:14:11.673359 19810 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:11.673521 19810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:11.673848 19810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:11.677208 19810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "7a36b1bd2a60435e947385b08b4ffa74"
format_stamp: "Formatted at 2026-09-24 03:14:11 on dist-test-slave-5l98"
server_key: "edc3f52068236b0a5c812564136aff99"
server_key_iv: "fb9accc23b1f35bbddab8af56f50c547"
server_key_version: "encryptionkey@0"
I20260924 03:14:11.678126 19810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7a36b1bd2a60435e947385b08b4ffa74"
format_stamp: "Formatted at 2026-09-24 03:14:11 on dist-test-slave-5l98"
server_key: "edc3f52068236b0a5c812564136aff99"
server_key_iv: "fb9accc23b1f35bbddab8af56f50c547"
server_key_version: "encryptionkey@0"
I20260924 03:14:11.683844 19810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 03:14:11.687999 19825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:11.689803 19810 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260924 03:14:11.689988 19810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "7a36b1bd2a60435e947385b08b4ffa74"
format_stamp: "Formatted at 2026-09-24 03:14:11 on dist-test-slave-5l98"
server_key: "edc3f52068236b0a5c812564136aff99"
server_key_iv: "fb9accc23b1f35bbddab8af56f50c547"
server_key_version: "encryptionkey@0"
I20260924 03:14:11.690191 19810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:11.713349 19810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:11.718006 19810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:11.718299 19810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:11.719394 19810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:14:11.720840 19810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:14:11.720922 19810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:11.721010 19810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:14:11.721053 19810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:11.759634 19810 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42327
I20260924 03:14:11.759701 19937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42327 every 8 connection(s)
I20260924 03:14:11.761188 19810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:14:11.763312 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 19810
I20260924 03:14:11.763532 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:14:11.764109 388 external_mini_cluster.cc:1428] Setting key c7e9df0a4209412076ab0f4e3940d5b3
Sep 24 03:14:11 dist-test-slave-5l98 krb5kdc[19704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:14:11.785852 19940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:11.764244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46391 (local address 127.0.97.62:39447)
0924 03:14:11.764779 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:11.764786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:11.765965 (+ 1179us) server_negotiation.cc:407] Connection header received
0924 03:14:11.767896 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:11.767904 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:11.768001 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:11.768124 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:11.769874 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:11.771049 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:11.772867 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:11.773625 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:11.773826 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:11.778872 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:11.778883 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:11.778887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:11.778926 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:11.781739 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:11.782955 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:11.782960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:11.782962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:11.783095 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:11.783665 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:11.783668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:11.783670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:11.783946 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:11.784058 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:11.785295 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0924 03:14:11.785501 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":276,"threads_started":1}
I20260924 03:14:11.787513 19941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:11.764913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39447 (local address 127.0.97.1:46391)
0924 03:14:11.765794 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:11.765842 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:11.767604 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:11.768329 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:11.768337 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:11.769050 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:11.769661 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:11.769685 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:11.771228 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:11.771232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:11.772703 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:11.772718 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:11.773911 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:11.773916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:11.774191 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:11.775103 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:14:11.775140 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:11.778534 (+ 3394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:11.781956 (+ 3422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:11.781970 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:11.781994 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:11.782786 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:11.783221 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:11.783229 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:11.783236 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:11.783539 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:11.784085 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:11.784097 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:11.784643 (+ 546us) client_negotiation.cc:769] Sending connection context
0924 03:14:11.785104 (+ 461us) client_negotiation.cc:240] Negotiation successful
0924 03:14:11.785881 (+ 777us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":370,"threads_started":1}
I20260924 03:14:11.789680 19938 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39447
I20260924 03:14:11.790095 19938 heartbeater.cc:464] Registering TS with master...
I20260924 03:14:11.790921 19938 heartbeater.cc:511] Master 127.0.97.62:39447 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 03:14:11.793453 19749 ts_manager.cc:194] Registered new tserver with Master: 7a36b1bd2a60435e947385b08b4ffa74 (127.0.97.1:42327)
I20260924 03:14:11.794993 19749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 03:14:11.829408 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39447
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:12.175557 19944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:12.175940 19944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:12.176115 19944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:12.183804 19944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:12.183950 19944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:12.184134 19944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:14:12.196545 19944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:12.198359 19944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:12.200393 19944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:12.214851 19952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:12.214824 19949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:12.214797 19950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:12.215910 19944 server_base.cc:1034] running on GCE node
I20260924 03:14:12.216606 19944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:12.217568 19944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:12.218883 19944 hybrid_clock.cc:648] HybridClock initialized: now 1790219652218837 us; error 33 us; skew 500 ppm
I20260924 03:14:12.219249 19944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:12 dist-test-slave-5l98 krb5kdc[19704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:12.222895 19944 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 03:14:12.224840 19944 webserver.cc:460] Webserver started at http://127.0.97.2:38083/ using document root <none> and password file <none>
I20260924 03:14:12.225703 19944 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:12.225869 19944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:12.226270 19944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:12.230715 19944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "562bb918cd384e6191d99a64790743f9"
format_stamp: "Formatted at 2026-09-24 03:14:12 on dist-test-slave-5l98"
server_key: "d96b923997573f7a997a1047475cfe20"
server_key_iv: "f97106fe2a99eb89403a6c300498e9e0"
server_key_version: "encryptionkey@0"
I20260924 03:14:12.231665 19944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "562bb918cd384e6191d99a64790743f9"
format_stamp: "Formatted at 2026-09-24 03:14:12 on dist-test-slave-5l98"
server_key: "d96b923997573f7a997a1047475cfe20"
server_key_iv: "f97106fe2a99eb89403a6c300498e9e0"
server_key_version: "encryptionkey@0"
I20260924 03:14:12.237620 19944 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 03:14:12.241801 19959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:12.243413 19944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:14:12.243708 19944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "562bb918cd384e6191d99a64790743f9"
format_stamp: "Formatted at 2026-09-24 03:14:12 on dist-test-slave-5l98"
server_key: "d96b923997573f7a997a1047475cfe20"
server_key_iv: "f97106fe2a99eb89403a6c300498e9e0"
server_key_version: "encryptionkey@0"
I20260924 03:14:12.243919 19944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:12.283434 19944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:12.288807 19944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:12.289160 19944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:12.290203 19944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:14:12.291677 19944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:14:12.291759 19944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:12.291940 19944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:14:12.292002 19944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:12.327992 19944 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36611
I20260924 03:14:12.328097 20071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36611 every 8 connection(s)
I20260924 03:14:12.329348 19944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:14:12.336824 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 19944
I20260924 03:14:12.337033 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:14:12.337606 388 external_mini_cluster.cc:1428] Setting key f341b813bd7d1550b3503a6d6d76d40a
Sep 24 03:14:12 dist-test-slave-5l98 krb5kdc[19704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:14:12.353425 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:12.331995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37661 (local address 127.0.97.62:39447)
0924 03:14:12.332537 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:12.332544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:12.333418 (+ 874us) server_negotiation.cc:407] Connection header received
0924 03:14:12.335347 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:12.335351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:12.335424 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:12.335522 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:12.337255 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.338335 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.340140 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.340928 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.341091 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:12.345206 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:12.345216 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:12.345224 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:12.345276 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:12.349308 (+ 4032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:12.350481 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:12.350486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:12.350488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:12.350634 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:12.351174 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:12.351182 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:12.351188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:12.351612 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:12.351774 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:12.352585 (+ 811us) server_negotiation.cc:299] Negotiation successful
0924 03:14:12.352879 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":255,"threads_started":1}
I20260924 03:14:12.354055 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:12.332356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39447 (local address 127.0.97.2:37661)
0924 03:14:12.333239 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:12.333281 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:12.334993 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:12.335753 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:12.335762 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:12.336326 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:12.337015 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:12.337054 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.338525 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.338532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:12.339972 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:12.339987 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.341211 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.341217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:12.341466 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:12.342261 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:14:12.342312 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:12.344922 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:12.349486 (+ 4564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:12.349498 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:12.349520 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:12.350286 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:12.350748 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:12.350751 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:12.350754 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:12.351005 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:12.351740 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:12.351746 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:12.352105 (+ 359us) client_negotiation.cc:769] Sending connection context
0924 03:14:12.352354 (+ 249us) client_negotiation.cc:240] Negotiation successful
0924 03:14:12.352851 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":361,"threads_started":1}
I20260924 03:14:12.355715 20072 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39447
I20260924 03:14:12.356066 20072 heartbeater.cc:464] Registering TS with master...
I20260924 03:14:12.357029 20072 heartbeater.cc:511] Master 127.0.97.62:39447 requested a full tablet report, sending...
I20260924 03:14:12.358901 19749 ts_manager.cc:194] Registered new tserver with Master: 562bb918cd384e6191d99a64790743f9 (127.0.97.2:36611)
I20260924 03:14:12.360181 19749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37661
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 03:14:12.405085 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39447
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:12.743637 20078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:12.743963 20078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:12.744138 20078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:12.751859 20078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:12.751989 20078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:12.752184 20078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:14:12.763238 20078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:12.764628 20078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:12.765903 20078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:12.779542 20083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:12.780107 20084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:12.780576 20086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:12.781024 20078 server_base.cc:1034] running on GCE node
I20260924 03:14:12.781834 20078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:12.782915 20078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:12.784195 20078 hybrid_clock.cc:648] HybridClock initialized: now 1790219652784126 us; error 57 us; skew 500 ppm
I20260924 03:14:12.784615 20078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:12 dist-test-slave-5l98 krb5kdc[19704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:12.787989 20078 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 03:14:12.789767 20078 webserver.cc:460] Webserver started at http://127.0.97.3:33511/ using document root <none> and password file <none>
I20260924 03:14:12.790484 20078 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:12.790634 20078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:12.790951 20078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:12.794126 20078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "93c34a4f32e64e2998a6c5ccb64ed663"
format_stamp: "Formatted at 2026-09-24 03:14:12 on dist-test-slave-5l98"
server_key: "e9dd9243286879bb1752737a8bb80acf"
server_key_iv: "47a1f25bc3845dba1424ab160ded13ca"
server_key_version: "encryptionkey@0"
I20260924 03:14:12.795017 20078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "93c34a4f32e64e2998a6c5ccb64ed663"
format_stamp: "Formatted at 2026-09-24 03:14:12 on dist-test-slave-5l98"
server_key: "e9dd9243286879bb1752737a8bb80acf"
server_key_iv: "47a1f25bc3845dba1424ab160ded13ca"
server_key_version: "encryptionkey@0"
I20260924 03:14:12.798466 19938 heartbeater.cc:503] Master 127.0.97.62:39447 was elected leader, sending a full tablet report...
I20260924 03:14:12.801043 20078 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 03:14:12.805094 20093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:12.806766 20078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:14:12.807024 20078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "93c34a4f32e64e2998a6c5ccb64ed663"
format_stamp: "Formatted at 2026-09-24 03:14:12 on dist-test-slave-5l98"
server_key: "e9dd9243286879bb1752737a8bb80acf"
server_key_iv: "47a1f25bc3845dba1424ab160ded13ca"
server_key_version: "encryptionkey@0"
I20260924 03:14:12.807284 20078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:12.833381 20078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:12.838902 20078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:12.839258 20078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:12.840442 20078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:14:12.842053 20078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:14:12.842185 20078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:12.842360 20078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:14:12.842447 20078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:12.882076 20078 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36417
I20260924 03:14:12.882140 20205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36417 every 8 connection(s)
I20260924 03:14:12.883476 20078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:14:12.892369 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 20078
I20260924 03:14:12.892627 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:14:12.893131 388 external_mini_cluster.cc:1428] Setting key c3f7b869024253913d785950a19220e5
Sep 24 03:14:12 dist-test-slave-5l98 krb5kdc[19704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:14:12.907717 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:12.886390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47661 (local address 127.0.97.62:39447)
0924 03:14:12.886873 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:12.886877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:12.887734 (+ 857us) server_negotiation.cc:407] Connection header received
0924 03:14:12.889926 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:12.889938 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:12.890051 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:12.890180 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:12.891814 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.892795 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.894891 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.895553 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.895685 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:12.900016 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:12.900023 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:12.900027 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:12.900070 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:12.903505 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:12.904768 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:12.904774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:12.904777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:12.905055 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:12.905668 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:12.905672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:12.905674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:12.906001 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:12.906142 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:12.907022 (+ 880us) server_negotiation.cc:299] Negotiation successful
0924 03:14:12.907342 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":256,"threads_started":1}
I20260924 03:14:12.908864 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:12.886753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39447 (local address 127.0.97.3:47661)
0924 03:14:12.887562 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:12.887599 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:12.889573 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:12.890473 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:12.890482 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:12.891036 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:12.891622 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:12.891641 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.892975 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.892980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:12.894594 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:12.894615 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.895841 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.895849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:12.896185 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:12.897103 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:14:12.897128 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:12.899590 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:12.903822 (+ 4232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:12.903832 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:12.903851 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:12.904430 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:12.905261 (+ 831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:12.905266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:12.905271 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:12.905528 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:12.906133 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:12.906139 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:12.906523 (+ 384us) client_negotiation.cc:769] Sending connection context
0924 03:14:12.906784 (+ 261us) client_negotiation.cc:240] Negotiation successful
0924 03:14:12.907246 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":291,"threads_started":1}
I20260924 03:14:12.910458 20206 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39447
I20260924 03:14:12.910772 20206 heartbeater.cc:464] Registering TS with master...
I20260924 03:14:12.911588 20206 heartbeater.cc:511] Master 127.0.97.62:39447 requested a full tablet report, sending...
I20260924 03:14:12.913316 19748 ts_manager.cc:194] Registered new tserver with Master: 93c34a4f32e64e2998a6c5ccb64ed663 (127.0.97.3:36417)
I20260924 03:14:12.914222 19748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47661
I20260924 03:14:12.922508 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:14:12.941251 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:12.927226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.0.97.62:39447)
0924 03:14:12.927405 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:12.927410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:12.927824 (+ 414us) server_negotiation.cc:407] Connection header received
0924 03:14:12.927938 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:12.927942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:12.928014 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:12.928110 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:12.929799 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.930576 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.932148 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.932680 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.932925 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:12.935522 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:12.935528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:12.935531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:12.935570 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:12.938497 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:12.939277 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:12.939281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:12.939284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:12.939415 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:12.939819 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:12.939822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:12.939824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:12.940164 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:12.940347 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:12.940792 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 03:14:12.941023 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:14:12.946103 19748 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:14:12.953471 19748 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:14:12.981118 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:12.967978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36611 (local address 127.0.0.1:58266)
0924 03:14:12.969930 (+ 1952us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:12.969956 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:12.970102 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:12.972357 (+ 2255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:12.972361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:12.972388 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:12.972785 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:12.972794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.975283 (+ 2489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.975291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:12.978169 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:12.978186 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.979837 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.979845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:12.980069 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:12.980135 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 03:14:12.980290 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 03:14:12.980571 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1761,"mutex_wait_us":30,"thread_start_us":1847,"threads_started":1}
I20260924 03:14:12.981519 20219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:12.967614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36417 (local address 127.0.0.1:47738)
0924 03:14:12.968807 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:12.968872 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:12.969106 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:12.971501 (+ 2395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:12.971511 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:12.971555 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:12.972039 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:12.972053 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.975291 (+ 3238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.975297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:12.978747 (+ 3450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:12.978765 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.980560 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.980567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:12.980758 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:12.980796 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 03:14:12.980868 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 03:14:12.981031 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":103,"thread_start_us":631,"threads_started":1}
I20260924 03:14:12.982129 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:12.969744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43142 (local address 127.0.97.1:42327)
0924 03:14:12.971097 (+ 1353us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:12.971106 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:12.971856 (+ 750us) server_negotiation.cc:407] Connection header received
0924 03:14:12.971986 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:12.971995 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:12.972249 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:12.972964 (+ 715us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:12.973902 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.975031 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.979053 (+ 4022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.980426 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.980630 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:12.980848 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:12.981614 (+ 766us) server_negotiation.cc:299] Negotiation successful
0924 03:14:12.981818 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1188,"thread_start_us":998,"threads_started":1}
I20260924 03:14:12.982262 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:12.968552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.0.97.2:36611)
0924 03:14:12.971871 (+ 3319us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:12.971878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:12.971901 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:14:12.971987 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:12.971993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:12.972163 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:12.972380 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:12.973034 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.975088 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.978349 (+ 3261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.979608 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.979747 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:12.979956 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:12.980392 (+ 436us) server_negotiation.cc:299] Negotiation successful
0924 03:14:12.980618 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3171,"thread_start_us":3079,"threads_started":1}
I20260924 03:14:12.982635 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:12.969103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42327 (local address 127.0.0.1:43142)
0924 03:14:12.971556 (+ 2453us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:12.971574 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:12.971703 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:12.973228 (+ 1525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:12.973235 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:12.973266 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:12.973700 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:12.973712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.975194 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.975203 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:12.978747 (+ 3544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:12.978765 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.981004 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.981014 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:12.981208 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:12.981240 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 03:14:12.981326 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 03:14:12.981460 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2311,"mutex_wait_us":361,"thread_start_us":2163,"threads_started":1}
I20260924 03:14:12.982946 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:12.967975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47738 (local address 127.0.97.3:36417)
0924 03:14:12.970710 (+ 2735us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:12.970723 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:12.970767 (+ 44us) server_negotiation.cc:407] Connection header received
0924 03:14:12.970881 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:12.970891 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:12.971203 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:12.972247 (+ 1044us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:12.972623 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.975172 (+ 2549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.978985 (+ 3813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:12.980294 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:12.982046 (+ 1752us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:12.982337 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:12.982412 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 03:14:12.982593 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2540,"thread_start_us":2392,"threads_started":1}
I20260924 03:14:12.986611 19873 tablet_service.cc:1467] Processing CreateTablet for tablet d98a54e7069043c4af34b867cffbcf7d (DEFAULT_TABLE table=test-table [id=9b0040d6a75b4a92bcdb2e02f761cef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:14:12.986611 20007 tablet_service.cc:1467] Processing CreateTablet for tablet d98a54e7069043c4af34b867cffbcf7d (DEFAULT_TABLE table=test-table [id=9b0040d6a75b4a92bcdb2e02f761cef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:14:12.986601 20141 tablet_service.cc:1467] Processing CreateTablet for tablet d98a54e7069043c4af34b867cffbcf7d (DEFAULT_TABLE table=test-table [id=9b0040d6a75b4a92bcdb2e02f761cef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:14:12.988062 20141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d98a54e7069043c4af34b867cffbcf7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:12.988176 19873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d98a54e7069043c4af34b867cffbcf7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:12.988173 20007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d98a54e7069043c4af34b867cffbcf7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:13.002115 20225 tablet_bootstrap.cc:492] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663: Bootstrap starting.
I20260924 03:14:13.004609 20227 tablet_bootstrap.cc:492] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9: Bootstrap starting.
I20260924 03:14:13.005445 20226 tablet_bootstrap.cc:492] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74: Bootstrap starting.
I20260924 03:14:13.006390 20225 tablet_bootstrap.cc:654] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:13.007865 20225 log.cc:824] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:13.009434 20226 tablet_bootstrap.cc:654] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:13.010869 20226 log.cc:824] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:13.011550 20225 tablet_bootstrap.cc:492] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663: No bootstrap required, opened a new log
I20260924 03:14:13.011901 20225 ts_tablet_manager.cc:1397] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260924 03:14:13.012419 20227 tablet_bootstrap.cc:654] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:13.013971 20226 tablet_bootstrap.cc:492] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74: No bootstrap required, opened a new log
I20260924 03:14:13.013991 20227 log.cc:824] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:13.014253 20226 ts_tablet_manager.cc:1397] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 03:14:13.017482 20227 tablet_bootstrap.cc:492] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9: No bootstrap required, opened a new log
I20260924 03:14:13.017786 20227 ts_tablet_manager.cc:1397] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.001s
I20260924 03:14:13.022536 20226 raft_consensus.cc:348] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c34a4f32e64e2998a6c5ccb64ed663" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36417 } } peers { permanent_uuid: "562bb918cd384e6191d99a64790743f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36611 } } peers { permanent_uuid: "7a36b1bd2a60435e947385b08b4ffa74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42327 } }
I20260924 03:14:13.022949 20226 raft_consensus.cc:374] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:13.023072 20226 raft_consensus.cc:729] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a36b1bd2a60435e947385b08b4ffa74, State: Initialized, Role: FOLLOWER
I20260924 03:14:13.023283 20225 raft_consensus.cc:348] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c34a4f32e64e2998a6c5ccb64ed663" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36417 } } peers { permanent_uuid: "562bb918cd384e6191d99a64790743f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36611 } } peers { permanent_uuid: "7a36b1bd2a60435e947385b08b4ffa74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42327 } }
I20260924 03:14:13.023765 20225 raft_consensus.cc:374] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:13.023866 20225 raft_consensus.cc:729] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c34a4f32e64e2998a6c5ccb64ed663, State: Initialized, Role: FOLLOWER
I20260924 03:14:13.023763 20226 consensus_queue.cc:260] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c34a4f32e64e2998a6c5ccb64ed663" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36417 } } peers { permanent_uuid: "562bb918cd384e6191d99a64790743f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36611 } } peers { permanent_uuid: "7a36b1bd2a60435e947385b08b4ffa74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42327 } }
I20260924 03:14:13.024478 20225 consensus_queue.cc:260] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c34a4f32e64e2998a6c5ccb64ed663" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36417 } } peers { permanent_uuid: "562bb918cd384e6191d99a64790743f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36611 } } peers { permanent_uuid: "7a36b1bd2a60435e947385b08b4ffa74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42327 } }
I20260924 03:14:13.025964 20226 ts_tablet_manager.cc:1428] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74: Time spent starting tablet: real 0.012s user 0.007s sys 0.006s
I20260924 03:14:13.026935 20206 heartbeater.cc:503] Master 127.0.97.62:39447 was elected leader, sending a full tablet report...
I20260924 03:14:13.027681 20225 ts_tablet_manager.cc:1428] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260924 03:14:13.029247 20227 raft_consensus.cc:348] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c34a4f32e64e2998a6c5ccb64ed663" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36417 } } peers { permanent_uuid: "562bb918cd384e6191d99a64790743f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36611 } } peers { permanent_uuid: "7a36b1bd2a60435e947385b08b4ffa74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42327 } }
I20260924 03:14:13.029661 20227 raft_consensus.cc:374] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:13.029790 20227 raft_consensus.cc:729] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb918cd384e6191d99a64790743f9, State: Initialized, Role: FOLLOWER
I20260924 03:14:13.030452 20227 consensus_queue.cc:260] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c34a4f32e64e2998a6c5ccb64ed663" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36417 } } peers { permanent_uuid: "562bb918cd384e6191d99a64790743f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36611 } } peers { permanent_uuid: "7a36b1bd2a60435e947385b08b4ffa74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42327 } }
I20260924 03:14:13.031599 20072 heartbeater.cc:503] Master 127.0.97.62:39447 was elected leader, sending a full tablet report...
I20260924 03:14:13.032217 20227 ts_tablet_manager.cc:1428] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
W20260924 03:14:13.081621 20073 tablet.cc:2369] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:14:13.135514 20207 tablet.cc:2369] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:14:13.264106 19939 tablet.cc:2369] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:14:13.326977 20233 raft_consensus.cc:482] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:14:13.327454 20233 raft_consensus.cc:504] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c34a4f32e64e2998a6c5ccb64ed663" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36417 } } peers { permanent_uuid: "562bb918cd384e6191d99a64790743f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36611 } } peers { permanent_uuid: "7a36b1bd2a60435e947385b08b4ffa74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42327 } }
I20260924 03:14:13.330202 20233 leader_election.cc:288] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93c34a4f32e64e2998a6c5ccb64ed663 (127.0.97.3:36417), 7a36b1bd2a60435e947385b08b4ffa74 (127.0.97.1:42327)
I20260924 03:14:13.338975 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:13.330621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52087 (local address 127.0.97.3:36417)
0924 03:14:13.330859 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:13.330867 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:13.331648 (+ 781us) server_negotiation.cc:407] Connection header received
0924 03:14:13.332141 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:13.332150 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:13.332253 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:13.332327 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:13.333353 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:13.334591 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:13.336714 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:13.337891 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:13.338053 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:13.338167 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:13.338543 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 03:14:13.338694 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 03:14:13.340029 20234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:13.330695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36417 (local address 127.0.97.2:52087)
0924 03:14:13.331546 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:13.331567 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:13.331886 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:13.332626 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:13.332633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:13.332665 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:13.333170 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:13.333184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:13.334761 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:13.334769 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:13.336513 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:13.336556 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:13.338126 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:13.338132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:13.338348 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:13.338379 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 03:14:13.339388 (+ 1009us) client_negotiation.cc:240] Negotiation successful
0924 03:14:13.339565 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":480,"threads_started":1}
I20260924 03:14:13.340191 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:13.331685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42327 (local address 127.0.97.2:47541)
0924 03:14:13.332672 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:13.332693 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:13.332924 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:13.333351 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:13.333355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:13.333379 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:13.333733 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:13.333741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:13.334807 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:13.334814 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:13.337430 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:13.337449 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:13.339163 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:13.339171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:13.339428 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:13.339463 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 03:14:13.339565 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 03:14:13.339708 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":3,"thread_start_us":602,"threads_started":1}
I20260924 03:14:13.340056 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:13.331896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47541 (local address 127.0.97.1:42327)
0924 03:14:13.332122 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:13.332129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:13.332804 (+ 675us) server_negotiation.cc:407] Connection header received
0924 03:14:13.333128 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:13.333136 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:13.333233 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:13.333322 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:13.333886 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:13.334645 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:13.337651 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:13.338873 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:13.339062 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:13.339323 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:13.339628 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 03:14:13.339793 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:14:13.341289 20161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98a54e7069043c4af34b867cffbcf7d" candidate_uuid: "562bb918cd384e6191d99a64790743f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93c34a4f32e64e2998a6c5ccb64ed663" is_pre_election: true
I20260924 03:14:13.341445 19893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98a54e7069043c4af34b867cffbcf7d" candidate_uuid: "562bb918cd384e6191d99a64790743f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a36b1bd2a60435e947385b08b4ffa74" is_pre_election: true
I20260924 03:14:13.341848 20161 raft_consensus.cc:2454] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb918cd384e6191d99a64790743f9 in term 0.
I20260924 03:14:13.342070 19893 raft_consensus.cc:2454] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb918cd384e6191d99a64790743f9 in term 0.
I20260924 03:14:13.342656 19961 leader_election.cc:302] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb918cd384e6191d99a64790743f9, 93c34a4f32e64e2998a6c5ccb64ed663; no voters:
I20260924 03:14:13.343130 20233 raft_consensus.cc:2781] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:14:13.343303 20233 raft_consensus.cc:482] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:14:13.343439 20233 raft_consensus.cc:3037] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:13.346844 20233 raft_consensus.cc:504] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c34a4f32e64e2998a6c5ccb64ed663" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36417 } } peers { permanent_uuid: "562bb918cd384e6191d99a64790743f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36611 } } peers { permanent_uuid: "7a36b1bd2a60435e947385b08b4ffa74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42327 } }
I20260924 03:14:13.347609 20233 leader_election.cc:288] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 [CANDIDATE]: Term 1 election: Requested vote from peers 93c34a4f32e64e2998a6c5ccb64ed663 (127.0.97.3:36417), 7a36b1bd2a60435e947385b08b4ffa74 (127.0.97.1:42327)
I20260924 03:14:13.348232 20161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98a54e7069043c4af34b867cffbcf7d" candidate_uuid: "562bb918cd384e6191d99a64790743f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93c34a4f32e64e2998a6c5ccb64ed663"
I20260924 03:14:13.348395 19893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98a54e7069043c4af34b867cffbcf7d" candidate_uuid: "562bb918cd384e6191d99a64790743f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a36b1bd2a60435e947385b08b4ffa74"
I20260924 03:14:13.348474 20161 raft_consensus.cc:3037] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:13.348698 19893 raft_consensus.cc:3037] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:13.350979 20161 raft_consensus.cc:2454] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb918cd384e6191d99a64790743f9 in term 1.
I20260924 03:14:13.351502 19893 raft_consensus.cc:2454] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb918cd384e6191d99a64790743f9 in term 1.
I20260924 03:14:13.351524 19961 leader_election.cc:302] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb918cd384e6191d99a64790743f9, 93c34a4f32e64e2998a6c5ccb64ed663; no voters:
I20260924 03:14:13.351938 20233 raft_consensus.cc:2781] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:13.352571 20233 raft_consensus.cc:686] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 [term 1 LEADER]: Becoming Leader. State: Replica: 562bb918cd384e6191d99a64790743f9, State: Running, Role: LEADER
I20260924 03:14:13.353314 20233 consensus_queue.cc:237] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c34a4f32e64e2998a6c5ccb64ed663" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36417 } } peers { permanent_uuid: "562bb918cd384e6191d99a64790743f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36611 } } peers { permanent_uuid: "7a36b1bd2a60435e947385b08b4ffa74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42327 } }
I20260924 03:14:13.358625 19749 catalog_manager.cc:5495] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 562bb918cd384e6191d99a64790743f9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "562bb918cd384e6191d99a64790743f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c34a4f32e64e2998a6c5ccb64ed663" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb918cd384e6191d99a64790743f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a36b1bd2a60435e947385b08b4ffa74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42327 } health_report { overall_health: UNKNOWN } } }
I20260924 03:14:13.440418 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:13.434328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58268 (local address 127.0.97.2:36611)
0924 03:14:13.434519 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:13.434525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:13.434541 (+ 16us) server_negotiation.cc:407] Connection header received
0924 03:14:13.434628 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:13.434631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:13.434729 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:13.434857 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:14:13.435257 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:13.436153 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:13.437807 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:13.438257 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:13.438355 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:13.439156 (+ 801us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:14:13.439832 (+ 676us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:14:13.439953 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:13.440100 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 03:14:13.440176 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 03:14:13.449405 20161 raft_consensus.cc:1261] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663 [term 1 FOLLOWER]: Refusing update from remote peer 562bb918cd384e6191d99a64790743f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:14:13.449411 19893 raft_consensus.cc:1261] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74 [term 1 FOLLOWER]: Refusing update from remote peer 562bb918cd384e6191d99a64790743f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:14:13.450383 20233 consensus_queue.cc:1035] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36b1bd2a60435e947385b08b4ffa74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:14:13.450750 20236 consensus_queue.cc:1035] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93c34a4f32e64e2998a6c5ccb64ed663" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:14:13.476650 19749 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:14:13.476943 19749 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:14:13.480665 19749 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 243f8160e6894cd4aca1866d944ad637: Sending DeleteTablet for 3 replicas of tablet d98a54e7069043c4af34b867cffbcf7d
I20260924 03:14:13.482076 20007 tablet_service.cc:1514] Processing DeleteTablet for tablet d98a54e7069043c4af34b867cffbcf7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:14:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58266
I20260924 03:14:13.482416 19873 tablet_service.cc:1514] Processing DeleteTablet for tablet d98a54e7069043c4af34b867cffbcf7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:14:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43142
I20260924 03:14:13.482991 20141 tablet_service.cc:1514] Processing DeleteTablet for tablet d98a54e7069043c4af34b867cffbcf7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:14:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47738
I20260924 03:14:13.483698 20250 tablet_replica.cc:330] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74: stopping tablet replica
I20260924 03:14:13.484293 20250 raft_consensus.cc:2229] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:14:13.484997 20250 raft_consensus.cc:2258] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:14:13.485069 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 19810
I20260924 03:14:13.487687 20250 ts_tablet_manager.cc:1905] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:14:13.493829 20249 tablet_replica.cc:330] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9: stopping tablet replica
I20260924 03:14:13.494516 20249 raft_consensus.cc:2229] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:14:13.495404 20249 raft_consensus.cc:2258] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:14:13.497761 20249 ts_tablet_manager.cc:1905] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:14:13.498096 20250 ts_tablet_manager.cc:1918] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:14:13.498443 20250 log.cc:1196] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-0/wal/wals/d98a54e7069043c4af34b867cffbcf7d
I20260924 03:14:13.499462 20250 ts_tablet_manager.cc:1939] T d98a54e7069043c4af34b867cffbcf7d P 7a36b1bd2a60435e947385b08b4ffa74: Deleting consensus metadata
I20260924 03:14:13.501979 19734 catalog_manager.cc:4841] TS 7a36b1bd2a60435e947385b08b4ffa74 (127.0.97.1:42327): tablet d98a54e7069043c4af34b867cffbcf7d (table test-table [id=9b0040d6a75b4a92bcdb2e02f761cef6]) successfully deleted
I20260924 03:14:13.505772 20249 ts_tablet_manager.cc:1918] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:14:13.505954 20249 log.cc:1196] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/d98a54e7069043c4af34b867cffbcf7d
I20260924 03:14:13.506678 20249 ts_tablet_manager.cc:1939] T d98a54e7069043c4af34b867cffbcf7d P 562bb918cd384e6191d99a64790743f9: Deleting consensus metadata
I20260924 03:14:13.508764 19733 catalog_manager.cc:4841] TS 562bb918cd384e6191d99a64790743f9 (127.0.97.2:36611): tablet d98a54e7069043c4af34b867cffbcf7d (table test-table [id=9b0040d6a75b4a92bcdb2e02f761cef6]) successfully deleted
I20260924 03:14:13.509022 20251 tablet_replica.cc:330] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663: stopping tablet replica
I20260924 03:14:13.509860 20251 raft_consensus.cc:2229] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:14:13.510455 20251 raft_consensus.cc:2258] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:14:13.512609 20251 ts_tablet_manager.cc:1905] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 03:14:13 dist-test-slave-5l98 krb5kdc[19704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 03:14:13.520684 20251 ts_tablet_manager.cc:1918] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:14:13.520888 20251 log.cc:1196] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/d98a54e7069043c4af34b867cffbcf7d
I20260924 03:14:13.521447 20251 ts_tablet_manager.cc:1939] T d98a54e7069043c4af34b867cffbcf7d P 93c34a4f32e64e2998a6c5ccb64ed663: Deleting consensus metadata
I20260924 03:14:13.523473 19734 catalog_manager.cc:4841] TS 93c34a4f32e64e2998a6c5ccb64ed663 (127.0.97.3:36417): tablet d98a54e7069043c4af34b867cffbcf7d (table test-table [id=9b0040d6a75b4a92bcdb2e02f761cef6]) successfully deleted
I20260924 03:14:13.524622 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:13.486565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43150 (local address 127.0.97.1:42327)
0924 03:14:13.486745 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:13.486751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:13.506641 (+ 19890us) server_negotiation.cc:407] Connection header received
0924 03:14:13.506999 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:13.507008 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:13.507144 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:13.507258 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:13.509616 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:13.510499 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:13.512355 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:13.512968 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:13.513115 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:13.517603 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:13.517620 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:13.517635 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:13.517676 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:13.521035 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:13.521920 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:13.521928 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:13.521931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:13.522065 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:13.522489 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:13.522493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:13.522496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:13.522873 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:13.523209 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:13.523783 (+ 574us) server_negotiation.cc:299] Negotiation successful
0924 03:14:13.524042 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:14:13.525578 19933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:13.618943 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 19810
I20260924 03:14:13.638224 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 19944
Sep 24 03:14:13 dist-test-slave-5l98 krb5kdc[19704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 03:14:13.657094 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:13.639067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58270 (local address 127.0.97.2:36611)
0924 03:14:13.639291 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:13.639298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:13.639373 (+ 75us) server_negotiation.cc:407] Connection header received
0924 03:14:13.639448 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:13.639454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:13.639546 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:13.639686 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:13.641071 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:13.642413 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:13.643876 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:13.644719 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:13.645006 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:13.649621 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:13.649662 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:13.649685 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:13.649739 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:13.653844 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:13.654644 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:13.654656 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:13.654659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:13.654830 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:13.655311 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:13.655317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:13.655321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:13.655803 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:13.656159 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:13.656413 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 03:14:13.656688 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 03:14:13.657963 20067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:13.776417 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 19944
I20260924 03:14:13.796756 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 20078
Sep 24 03:14:13 dist-test-slave-5l98 krb5kdc[19704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 03:14:13.813817 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:13.797710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47746 (local address 127.0.97.3:36417)
0924 03:14:13.797982 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:13.797989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:13.798011 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:14:13.798279 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:13.798283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:13.798373 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:13.798492 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:13.799984 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:13.800808 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:13.802148 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:13.802539 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:13.802686 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:13.807615 (+ 4929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:13.807632 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:13.807647 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:13.807688 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:13.810733 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:13.811587 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:13.811595 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:13.811597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:13.811746 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:13.812125 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:13.812130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:13.812134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:13.812602 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:13.812910 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:13.813227 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 03:14:13.813453 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260924 03:14:13.814572 20201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:13.907212 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 20078
I20260924 03:14:13.924244 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 19717
I20260924 03:14:13.925676 19779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:14.040146 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 19717
2026-09-24T03:14:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4835 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T03:14:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:14Z Disabled control of system clock
I20260924 03:14:14.120616 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37723
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:14:14.492726 20260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:14.493055 20260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:14.493206 20260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:14.501134 20260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:14.501266 20260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:14.501346 20260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:14.501441 20260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:14.501504 20260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:14.511287 20260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37723
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:14.512699 20260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:14.514462 20260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:14.526981 20265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:14.527051 20268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:14.527087 20266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:14.528324 20260 server_base.cc:1034] running on GCE node
I20260924 03:14:14.529273 20260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:14.530736 20260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:14.531970 20260 hybrid_clock.cc:648] HybridClock initialized: now 1790219654531920 us; error 39 us; skew 500 ppm
I20260924 03:14:14.532387 20260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:14:14.534838 20260 webserver.cc:460] Webserver started at http://127.0.97.62:40957/ using document root <none> and password file <none>
I20260924 03:14:14.535547 20260 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:14.535658 20260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:14.536001 20260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:14.539036 20260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "e353c9e32ded4e5db2b3e88c510018d3"
format_stamp: "Formatted at 2026-09-24 03:14:14 on dist-test-slave-5l98"
server_key: "6a97d07791101c3ab7ea33c71e745306"
server_key_iv: "65e537fb7e61edd20a5f202eed0615e3"
server_key_version: "encryptionkey@0"
I20260924 03:14:14.539872 20260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "e353c9e32ded4e5db2b3e88c510018d3"
format_stamp: "Formatted at 2026-09-24 03:14:14 on dist-test-slave-5l98"
server_key: "6a97d07791101c3ab7ea33c71e745306"
server_key_iv: "65e537fb7e61edd20a5f202eed0615e3"
server_key_version: "encryptionkey@0"
I20260924 03:14:14.545647 20260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260924 03:14:14.549507 20274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:14.551113 20260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:14:14.551332 20260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "e353c9e32ded4e5db2b3e88c510018d3"
format_stamp: "Formatted at 2026-09-24 03:14:14 on dist-test-slave-5l98"
server_key: "6a97d07791101c3ab7ea33c71e745306"
server_key_iv: "65e537fb7e61edd20a5f202eed0615e3"
server_key_version: "encryptionkey@0"
I20260924 03:14:14.551579 20260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:14.576359 20260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:14.577392 20260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:14.577708 20260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:14.597226 20325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37723 every 8 connection(s)
I20260924 03:14:14.597229 20260 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37723
I20260924 03:14:14.598600 20260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:14.602674 20326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:14.607757 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 20260
I20260924 03:14:14.607999 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:14.608448 388 external_mini_cluster.cc:1428] Setting key 40bdfa5dbb3a36109dc019ed345e792c
I20260924 03:14:14.614110 20326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3: Bootstrap starting.
I20260924 03:14:14.617273 20326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:14.618302 20326 log.cc:824] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:14.621338 20326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3: No bootstrap required, opened a new log
I20260924 03:14:14.622340 20329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:14.610574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52192 (local address 127.0.97.62:37723)
0924 03:14:14.611648 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:14.611661 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:14.611703 (+ 42us) server_negotiation.cc:407] Connection header received
0924 03:14:14.613528 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:14.613553 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:14.614017 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:14.614447 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:14.614990 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:14.616277 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:14.617832 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:14.618580 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:14.618773 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:14.619986 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:14.619994 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:14.620008 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:14.620609 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:14.620688 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:14.620709 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:14.620852 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:14.621060 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 03:14:14.621377 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":373,"threads_started":1}
I20260924 03:14:14.630304 20326 raft_consensus.cc:348] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e353c9e32ded4e5db2b3e88c510018d3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37723 } }
I20260924 03:14:14.630736 20326 raft_consensus.cc:374] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:14.630856 20326 raft_consensus.cc:729] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e353c9e32ded4e5db2b3e88c510018d3, State: Initialized, Role: FOLLOWER
I20260924 03:14:14.631482 20326 consensus_queue.cc:260] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e353c9e32ded4e5db2b3e88c510018d3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37723 } }
I20260924 03:14:14.631743 20326 raft_consensus.cc:388] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:14.631901 20326 raft_consensus.cc:482] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:14.632081 20326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:14.634627 20326 raft_consensus.cc:504] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e353c9e32ded4e5db2b3e88c510018d3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37723 } }
I20260924 03:14:14.635154 20326 leader_election.cc:302] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e353c9e32ded4e5db2b3e88c510018d3; no voters:
I20260924 03:14:14.635761 20326 leader_election.cc:288] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:14.635946 20331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:14.636741 20331 raft_consensus.cc:686] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3 [term 1 LEADER]: Becoming Leader. State: Replica: e353c9e32ded4e5db2b3e88c510018d3, State: Running, Role: LEADER
I20260924 03:14:14.637303 20331 consensus_queue.cc:237] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e353c9e32ded4e5db2b3e88c510018d3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37723 } }
I20260924 03:14:14.638002 20326 sys_catalog.cc:564] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:14.640151 20332 sys_catalog.cc:455] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e353c9e32ded4e5db2b3e88c510018d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e353c9e32ded4e5db2b3e88c510018d3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37723 } } }
I20260924 03:14:14.640527 20332 sys_catalog.cc:458] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:14.644788 20333 sys_catalog.cc:455] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e353c9e32ded4e5db2b3e88c510018d3. Latest consensus state: current_term: 1 leader_uuid: "e353c9e32ded4e5db2b3e88c510018d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e353c9e32ded4e5db2b3e88c510018d3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37723 } } }
I20260924 03:14:14.645493 20333 sys_catalog.cc:458] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:14.646698 20337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:14.652308 20337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:14.662163 20337 catalog_manager.cc:1347] Generated new cluster ID: c76dcff0587642a98b7e35cdbfd37b92
I20260924 03:14:14.662310 20337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:14.685142 20337 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:14.686527 20337 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:14.696410 20337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3: Generated new TSK 0
I20260924 03:14:14.697409 20337 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:14:14.712226 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37723
--builtin_ntp_servers=127.0.97.20:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:14:15.081828 20350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:15.082130 20350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:15.082273 20350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:15.090005 20350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:15.090123 20350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:15.090307 20350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 03:14:15.099766 20350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:15.101163 20350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:15.102994 20350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:15.116945 20356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:15.117210 20355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:15.117861 20358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:15.118314 20350 server_base.cc:1034] running on GCE node
I20260924 03:14:15.118968 20350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:15.120110 20350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:15.121384 20350 hybrid_clock.cc:648] HybridClock initialized: now 1790219655121297 us; error 47 us; skew 500 ppm
I20260924 03:14:15.121840 20350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:14:15.124419 20350 webserver.cc:460] Webserver started at http://127.0.97.1:37683/ using document root <none> and password file <none>
I20260924 03:14:15.125204 20350 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:15.125298 20350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:15.125669 20350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:15.128687 20350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-0/data/instance:
uuid: "4af7dafec112490490274030f7f9ee57"
format_stamp: "Formatted at 2026-09-24 03:14:15 on dist-test-slave-5l98"
server_key: "d12e910ff50cea36ba51a9d232946cfb"
server_key_iv: "c546d39be520899d17eaaeed4be8f1ca"
server_key_version: "encryptionkey@0"
I20260924 03:14:15.129544 20350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4af7dafec112490490274030f7f9ee57"
format_stamp: "Formatted at 2026-09-24 03:14:15 on dist-test-slave-5l98"
server_key: "d12e910ff50cea36ba51a9d232946cfb"
server_key_iv: "c546d39be520899d17eaaeed4be8f1ca"
server_key_version: "encryptionkey@0"
I20260924 03:14:15.135026 20350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:14:15.139036 20364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:15.140709 20350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:14:15.140903 20350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-0/wal
uuid: "4af7dafec112490490274030f7f9ee57"
format_stamp: "Formatted at 2026-09-24 03:14:15 on dist-test-slave-5l98"
server_key: "d12e910ff50cea36ba51a9d232946cfb"
server_key_iv: "c546d39be520899d17eaaeed4be8f1ca"
server_key_version: "encryptionkey@0"
I20260924 03:14:15.141158 20350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:15.158277 20350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:15.159184 20350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:15.159487 20350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:15.160553 20350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:14:15.162176 20350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:14:15.162274 20350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:15.162411 20350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:14:15.162500 20350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:15.196002 20350 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43665
I20260924 03:14:15.196125 20476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43665 every 8 connection(s)
I20260924 03:14:15.197499 20350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-0/data/info.pb
I20260924 03:14:15.203166 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 20350
I20260924 03:14:15.203343 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-0/wal/instance
I20260924 03:14:15.203795 388 external_mini_cluster.cc:1428] Setting key fb04bb25df26c01c907b83f818be46d1
I20260924 03:14:15.206817 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37723
--builtin_ntp_servers=127.0.97.20:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:14:15.211385 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:15.200383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42853 (local address 127.0.97.62:37723)
0924 03:14:15.200863 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:15.200869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:15.202109 (+ 1240us) server_negotiation.cc:407] Connection header received
0924 03:14:15.204140 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:15.204145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:15.204239 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:15.204377 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:15.206086 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:15.206829 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:15.208437 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:15.208899 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:15.209034 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:15.210486 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:15.210491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:15.210494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:15.210512 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:15.210551 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:15.210581 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:15.210680 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:15.210917 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 03:14:15.211027 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":251,"threads_started":1}
I20260924 03:14:15.212414 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:15.201040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37723 (local address 127.0.97.1:42853)
0924 03:14:15.201952 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:15.201988 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:15.203819 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:15.204808 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:15.204828 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:15.204967 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:15.205885 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:15.205910 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:15.206980 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:15.206986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:15.208284 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:15.208296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:15.209145 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:15.209152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:15.209397 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:15.210135 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:14:15.210138 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:15.210184 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:14:15.210266 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:15.210677 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:15.210690 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:15.210707 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 03:14:15.210909 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 03:14:15.211334 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":276,"threads_started":1}
I20260924 03:14:15.214125 20477 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37723
I20260924 03:14:15.214470 20477 heartbeater.cc:464] Registering TS with master...
I20260924 03:14:15.215243 20477 heartbeater.cc:511] Master 127.0.97.62:37723 requested a full tablet report, sending...
I20260924 03:14:15.217365 20291 ts_manager.cc:194] Registered new tserver with Master: 4af7dafec112490490274030f7f9ee57 (127.0.97.1:43665)
I20260924 03:14:15.218888 20291 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:42853
W20260924 03:14:15.553798 20481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:15.554152 20481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:15.554311 20481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:15.562032 20481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:15.562223 20481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:15.562422 20481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 03:14:15.572726 20481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:15.574098 20481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:15.575923 20481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:15.589938 20486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:15.590052 20487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:15.590497 20481 server_base.cc:1034] running on GCE node
W20260924 03:14:15.590832 20489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:15.591495 20481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:15.592674 20481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:15.593961 20481 hybrid_clock.cc:648] HybridClock initialized: now 1790219655593886 us; error 68 us; skew 500 ppm
I20260924 03:14:15.594329 20481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:14:15.597167 20481 webserver.cc:460] Webserver started at http://127.0.97.2:33429/ using document root <none> and password file <none>
I20260924 03:14:15.598061 20481 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:15.598191 20481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:15.598575 20481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:15.602707 20481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-1/data/instance:
uuid: "d150d8a6b6914c10b8aabc479a4e0067"
format_stamp: "Formatted at 2026-09-24 03:14:15 on dist-test-slave-5l98"
server_key: "5b291c01aea89a7e64246009201eab72"
server_key_iv: "2b707818db52988c1056e9583e5bc8a8"
server_key_version: "encryptionkey@0"
I20260924 03:14:15.603801 20481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d150d8a6b6914c10b8aabc479a4e0067"
format_stamp: "Formatted at 2026-09-24 03:14:15 on dist-test-slave-5l98"
server_key: "5b291c01aea89a7e64246009201eab72"
server_key_iv: "2b707818db52988c1056e9583e5bc8a8"
server_key_version: "encryptionkey@0"
I20260924 03:14:15.610780 20481 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260924 03:14:15.615051 20495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:15.616684 20481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 03:14:15.616870 20481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-1/wal
uuid: "d150d8a6b6914c10b8aabc479a4e0067"
format_stamp: "Formatted at 2026-09-24 03:14:15 on dist-test-slave-5l98"
server_key: "5b291c01aea89a7e64246009201eab72"
server_key_iv: "2b707818db52988c1056e9583e5bc8a8"
server_key_version: "encryptionkey@0"
I20260924 03:14:15.617110 20481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:15.632910 20481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:15.633795 20481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:15.634088 20481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:15.635111 20481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:14:15.636781 20481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:14:15.636875 20481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:15.637037 20481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:14:15.637120 20481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:15.672767 20481 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38219
I20260924 03:14:15.672976 20607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38219 every 8 connection(s)
I20260924 03:14:15.674171 20481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-1/data/info.pb
I20260924 03:14:15.674535 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 20481
I20260924 03:14:15.674716 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-1/wal/instance
I20260924 03:14:15.675390 388 external_mini_cluster.cc:1428] Setting key 7103362b8482b0544e0e4a230a348158
I20260924 03:14:15.678912 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37723
--builtin_ntp_servers=127.0.97.20:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:14:15.691107 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:15.677412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59989 (local address 127.0.97.62:37723)
0924 03:14:15.677608 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:15.677614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:15.679248 (+ 1634us) server_negotiation.cc:407] Connection header received
0924 03:14:15.682196 (+ 2948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:15.682201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:15.682267 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:15.682362 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:15.685358 (+ 2996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:15.686106 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:15.687700 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:15.688364 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:15.688543 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:15.690044 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:15.690053 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:15.690061 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:15.690088 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:15.690152 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:15.690171 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:15.690270 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:15.690585 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 03:14:15.690713 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 03:14:15.692065 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:15.677925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37723 (local address 127.0.97.2:59989)
0924 03:14:15.679057 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:15.679113 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:15.681882 (+ 2769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:15.682636 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:15.682650 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:15.682773 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:15.685173 (+ 2400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:15.685192 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:15.686291 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:15.686296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:15.687527 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:15.687539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:15.688671 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:15.688677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:15.688914 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:15.689691 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:14:15.689694 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:15.689738 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:14:15.689825 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:15.690342 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:15.690351 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:15.690359 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 03:14:15.690526 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 03:14:15.690852 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":370,"threads_started":1}
I20260924 03:14:15.693809 20608 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37723
I20260924 03:14:15.694141 20608 heartbeater.cc:464] Registering TS with master...
I20260924 03:14:15.695000 20608 heartbeater.cc:511] Master 127.0.97.62:37723 requested a full tablet report, sending...
I20260924 03:14:15.696745 20291 ts_manager.cc:194] Registered new tserver with Master: d150d8a6b6914c10b8aabc479a4e0067 (127.0.97.2:38219)
I20260924 03:14:15.697633 20291 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:59989
W20260924 03:14:16.022625 20611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:16.022961 20611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:16.023118 20611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:16.030715 20611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:16.030827 20611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:16.031008 20611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 03:14:16.040562 20611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:16.041944 20611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:16.043685 20611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:16.057780 20616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:16.057843 20619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:16.057843 20617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:16.059507 20611 server_base.cc:1034] running on GCE node
I20260924 03:14:16.060187 20611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:16.061338 20611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:16.062592 20611 hybrid_clock.cc:648] HybridClock initialized: now 1790219656062554 us; error 57 us; skew 500 ppm
I20260924 03:14:16.062937 20611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:14:16.070416 20611 webserver.cc:460] Webserver started at http://127.0.97.3:42311/ using document root <none> and password file <none>
I20260924 03:14:16.071213 20611 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:16.071375 20611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:16.071712 20611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:16.074770 20611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-2/data/instance:
uuid: "83d05b0b2c0841aabc56c5f38daf964f"
format_stamp: "Formatted at 2026-09-24 03:14:16 on dist-test-slave-5l98"
server_key: "f46da72e6271711f1cf4f58af3141fde"
server_key_iv: "4ada58b610f5f2d528272221c0f07f40"
server_key_version: "encryptionkey@0"
I20260924 03:14:16.075598 20611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-2/wal/instance:
uuid: "83d05b0b2c0841aabc56c5f38daf964f"
format_stamp: "Formatted at 2026-09-24 03:14:16 on dist-test-slave-5l98"
server_key: "f46da72e6271711f1cf4f58af3141fde"
server_key_iv: "4ada58b610f5f2d528272221c0f07f40"
server_key_version: "encryptionkey@0"
I20260924 03:14:16.081034 20611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 03:14:16.084988 20625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:16.086529 20611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:14:16.086737 20611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-2/wal
uuid: "83d05b0b2c0841aabc56c5f38daf964f"
format_stamp: "Formatted at 2026-09-24 03:14:16 on dist-test-slave-5l98"
server_key: "f46da72e6271711f1cf4f58af3141fde"
server_key_iv: "4ada58b610f5f2d528272221c0f07f40"
server_key_version: "encryptionkey@0"
I20260924 03:14:16.086990 20611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:16.102613 20611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:16.103518 20611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:16.103839 20611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:16.104959 20611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:14:16.106607 20611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:14:16.106730 20611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:16.106896 20611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:14:16.106986 20611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:16.144269 20611 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40171
I20260924 03:14:16.144308 20737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40171 every 8 connection(s)
I20260924 03:14:16.145738 20611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-2/data/info.pb
I20260924 03:14:16.149001 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 20611
I20260924 03:14:16.149188 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-2/wal/instance
I20260924 03:14:16.149755 388 external_mini_cluster.cc:1428] Setting key de478d04485b5b3536dedfa0d93e35f4
I20260924 03:14:16.159902 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.148609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56345 (local address 127.0.97.62:37723)
0924 03:14:16.148851 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:16.148858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:16.150574 (+ 1716us) server_negotiation.cc:407] Connection header received
0924 03:14:16.153036 (+ 2462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:16.153040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:16.153110 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:16.153214 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:16.154460 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.155261 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.157134 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.157533 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.157629 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:16.159124 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:16.159128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:16.159131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:16.159151 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:16.159182 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:16.159193 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:16.159266 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:16.159486 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 03:14:16.159567 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 03:14:16.160982 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.149168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37723 (local address 127.0.97.3:56345)
0924 03:14:16.150370 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:16.150439 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:16.152683 (+ 2244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:16.153448 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:16.153457 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:16.153555 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:16.154253 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:16.154270 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.155396 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.155401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:16.156995 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:16.157008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.157735 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.157740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:16.158036 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:16.158800 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:14:16.158803 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:16.158851 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:14:16.158934 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:16.159292 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:16.159301 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:16.159306 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:14:16.159456 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 03:14:16.159839 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":343,"threads_started":1}
I20260924 03:14:16.162565 20738 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37723
I20260924 03:14:16.162926 20738 heartbeater.cc:464] Registering TS with master...
I20260924 03:14:16.163724 20738 heartbeater.cc:511] Master 127.0.97.62:37723 requested a full tablet report, sending...
I20260924 03:14:16.165441 20291 ts_manager.cc:194] Registered new tserver with Master: 83d05b0b2c0841aabc56c5f38daf964f (127.0.97.3:40171)
I20260924 03:14:16.166325 20291 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:56345
I20260924 03:14:16.178849 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:14:16.190881 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.183576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.0.97.62:37723)
0924 03:14:16.183906 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:16.183911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:16.184276 (+ 365us) server_negotiation.cc:407] Connection header received
0924 03:14:16.184608 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:16.184614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:16.184787 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:16.184887 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:16.185534 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.186358 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.187960 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.188396 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.188578 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:16.189819 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:16.189824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:16.189828 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:16.189847 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:16.189901 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:16.189917 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:16.190003 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:16.190380 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 03:14:16.190496 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 03:14:16.203155 20291 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:14:16.213569 20291 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:14:16.222494 20477 heartbeater.cc:503] Master 127.0.97.62:37723 was elected leader, sending a full tablet report...
I20260924 03:14:16.239715 20750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.227153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40171 (local address 127.0.0.1:40332)
0924 03:14:16.228308 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:16.228343 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:16.228559 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:16.230364 (+ 1805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:16.230372 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:16.230409 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:16.230825 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:16.230837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.233847 (+ 3010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.233855 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:16.236687 (+ 2832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:16.236704 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.238621 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.238629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:16.238845 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:16.238900 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 03:14:16.239044 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 03:14:16.239268 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":169,"thread_start_us":771,"threads_started":1}
I20260924 03:14:16.240411 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.227762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40332 (local address 127.0.97.3:40171)
0924 03:14:16.229851 (+ 2089us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:16.229858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:16.229885 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:14:16.229962 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:16.229968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:16.230151 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:16.230362 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:16.231050 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.233675 (+ 2625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.236939 (+ 3264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.238418 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.238566 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:16.238866 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:16.239114 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 03:14:16.239317 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1934,"thread_start_us":1769,"threads_started":1}
I20260924 03:14:16.240612 20751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.227547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38219 (local address 127.0.0.1:55358)
0924 03:14:16.229124 (+ 1577us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:16.229143 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:16.229282 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:16.231458 (+ 2176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:16.231463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:16.231486 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:16.231929 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:16.231942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.233615 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.233622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:16.237362 (+ 3740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:16.237375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.239636 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.239644 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:16.239900 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:16.239948 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 03:14:16.240043 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 03:14:16.240199 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1447,"mutex_wait_us":1000,"thread_start_us":1296,"threads_started":1}
I20260924 03:14:16.241539 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.227838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55358 (local address 127.0.97.2:38219)
0924 03:14:16.231021 (+ 3183us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:16.231028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:16.231050 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:14:16.231125 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:16.231130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:16.231287 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:16.231474 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:16.232133 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.233474 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.237577 (+ 4103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.239103 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.239239 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:16.239466 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:16.240952 (+ 1486us) server_negotiation.cc:299] Negotiation successful
0924 03:14:16.241162 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3015,"thread_start_us":2868,"threads_started":1}
I20260924 03:14:16.246102 20673 tablet_service.cc:1467] Processing CreateTablet for tablet 635df502380b4efd976e53fd37106e56 (DEFAULT_TABLE table=test-table [id=ff0bb9e05ee143d6b90248cbc2ed3818]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:14:16.246088 20543 tablet_service.cc:1467] Processing CreateTablet for tablet 635df502380b4efd976e53fd37106e56 (DEFAULT_TABLE table=test-table [id=ff0bb9e05ee143d6b90248cbc2ed3818]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:14:16.247560 20673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 635df502380b4efd976e53fd37106e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:16.247593 20543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 635df502380b4efd976e53fd37106e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:16.249328 20755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.229121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46752 (local address 127.0.97.1:43665)
0924 03:14:16.230031 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:16.230042 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:16.236956 (+ 6914us) server_negotiation.cc:407] Connection header received
0924 03:14:16.237304 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:16.237316 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:16.237617 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:16.237850 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:16.238406 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.242182 (+ 3776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.244846 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.247694 (+ 2848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.247812 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:16.248085 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:16.248735 (+ 650us) server_negotiation.cc:299] Negotiation successful
0924 03:14:16.248918 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":569,"threads_started":1}
I20260924 03:14:16.250058 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.228467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43665 (local address 127.0.0.1:46752)
0924 03:14:16.236790 (+ 8323us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:16.236807 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:16.237082 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:16.237826 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:16.237831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:16.237858 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:16.238237 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:16.238251 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.242358 (+ 4107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.242394 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:16.244597 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:16.244614 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.248284 (+ 3670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.248291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:16.248506 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:16.248544 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 03:14:16.249568 (+ 1024us) client_negotiation.cc:240] Negotiation successful
0924 03:14:16.249754 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8172,"thread_start_us":7866,"threads_started":1}
I20260924 03:14:16.254591 20412 tablet_service.cc:1467] Processing CreateTablet for tablet 635df502380b4efd976e53fd37106e56 (DEFAULT_TABLE table=test-table [id=ff0bb9e05ee143d6b90248cbc2ed3818]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:14:16.256290 20412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 635df502380b4efd976e53fd37106e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:16.263024 20757 tablet_bootstrap.cc:492] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f: Bootstrap starting.
I20260924 03:14:16.265375 20756 tablet_bootstrap.cc:492] T 635df502380b4efd976e53fd37106e56 P d150d8a6b6914c10b8aabc479a4e0067: Bootstrap starting.
I20260924 03:14:16.267279 20757 tablet_bootstrap.cc:654] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:16.268671 20757 log.cc:824] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:16.269199 20756 tablet_bootstrap.cc:654] T 635df502380b4efd976e53fd37106e56 P d150d8a6b6914c10b8aabc479a4e0067: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:16.270454 20756 log.cc:824] T 635df502380b4efd976e53fd37106e56 P d150d8a6b6914c10b8aabc479a4e0067: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:16.274554 20756 tablet_bootstrap.cc:492] T 635df502380b4efd976e53fd37106e56 P d150d8a6b6914c10b8aabc479a4e0067: No bootstrap required, opened a new log
I20260924 03:14:16.274878 20756 ts_tablet_manager.cc:1397] T 635df502380b4efd976e53fd37106e56 P d150d8a6b6914c10b8aabc479a4e0067: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 03:14:16.274920 20760 tablet_bootstrap.cc:492] T 635df502380b4efd976e53fd37106e56 P 4af7dafec112490490274030f7f9ee57: Bootstrap starting.
I20260924 03:14:16.278597 20757 tablet_bootstrap.cc:492] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f: No bootstrap required, opened a new log
I20260924 03:14:16.278867 20760 tablet_bootstrap.cc:654] T 635df502380b4efd976e53fd37106e56 P 4af7dafec112490490274030f7f9ee57: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:16.278918 20757 ts_tablet_manager.cc:1397] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260924 03:14:16.280159 20760 log.cc:824] T 635df502380b4efd976e53fd37106e56 P 4af7dafec112490490274030f7f9ee57: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:16.283931 20760 tablet_bootstrap.cc:492] T 635df502380b4efd976e53fd37106e56 P 4af7dafec112490490274030f7f9ee57: No bootstrap required, opened a new log
I20260924 03:14:16.284260 20760 ts_tablet_manager.cc:1397] T 635df502380b4efd976e53fd37106e56 P 4af7dafec112490490274030f7f9ee57: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 03:14:16.286099 20756 raft_consensus.cc:348] T 635df502380b4efd976e53fd37106e56 P d150d8a6b6914c10b8aabc479a4e0067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d05b0b2c0841aabc56c5f38daf964f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40171 } } peers { permanent_uuid: "d150d8a6b6914c10b8aabc479a4e0067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 } } peers { permanent_uuid: "4af7dafec112490490274030f7f9ee57" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43665 } }
I20260924 03:14:16.286621 20756 raft_consensus.cc:374] T 635df502380b4efd976e53fd37106e56 P d150d8a6b6914c10b8aabc479a4e0067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:16.286727 20756 raft_consensus.cc:729] T 635df502380b4efd976e53fd37106e56 P d150d8a6b6914c10b8aabc479a4e0067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d150d8a6b6914c10b8aabc479a4e0067, State: Initialized, Role: FOLLOWER
I20260924 03:14:16.287451 20757 raft_consensus.cc:348] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d05b0b2c0841aabc56c5f38daf964f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40171 } } peers { permanent_uuid: "d150d8a6b6914c10b8aabc479a4e0067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 } } peers { permanent_uuid: "4af7dafec112490490274030f7f9ee57" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43665 } }
I20260924 03:14:16.287600 20756 consensus_queue.cc:260] T 635df502380b4efd976e53fd37106e56 P d150d8a6b6914c10b8aabc479a4e0067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d05b0b2c0841aabc56c5f38daf964f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40171 } } peers { permanent_uuid: "d150d8a6b6914c10b8aabc479a4e0067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 } } peers { permanent_uuid: "4af7dafec112490490274030f7f9ee57" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43665 } }
I20260924 03:14:16.287834 20757 raft_consensus.cc:374] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:16.287925 20757 raft_consensus.cc:729] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83d05b0b2c0841aabc56c5f38daf964f, State: Initialized, Role: FOLLOWER
I20260924 03:14:16.288551 20757 consensus_queue.cc:260] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d05b0b2c0841aabc56c5f38daf964f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40171 } } peers { permanent_uuid: "d150d8a6b6914c10b8aabc479a4e0067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 } } peers { permanent_uuid: "4af7dafec112490490274030f7f9ee57" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43665 } }
I20260924 03:14:16.289121 20608 heartbeater.cc:503] Master 127.0.97.62:37723 was elected leader, sending a full tablet report...
I20260924 03:14:16.291630 20756 ts_tablet_manager.cc:1428] T 635df502380b4efd976e53fd37106e56 P d150d8a6b6914c10b8aabc479a4e0067: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
I20260924 03:14:16.293185 20738 heartbeater.cc:503] Master 127.0.97.62:37723 was elected leader, sending a full tablet report...
I20260924 03:14:16.293874 20757 ts_tablet_manager.cc:1428] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260924 03:14:16.296368 20760 raft_consensus.cc:348] T 635df502380b4efd976e53fd37106e56 P 4af7dafec112490490274030f7f9ee57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d05b0b2c0841aabc56c5f38daf964f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40171 } } peers { permanent_uuid: "d150d8a6b6914c10b8aabc479a4e0067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 } } peers { permanent_uuid: "4af7dafec112490490274030f7f9ee57" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43665 } }
I20260924 03:14:16.296857 20760 raft_consensus.cc:374] T 635df502380b4efd976e53fd37106e56 P 4af7dafec112490490274030f7f9ee57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:16.296978 20760 raft_consensus.cc:729] T 635df502380b4efd976e53fd37106e56 P 4af7dafec112490490274030f7f9ee57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af7dafec112490490274030f7f9ee57, State: Initialized, Role: FOLLOWER
I20260924 03:14:16.297657 20760 consensus_queue.cc:260] T 635df502380b4efd976e53fd37106e56 P 4af7dafec112490490274030f7f9ee57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d05b0b2c0841aabc56c5f38daf964f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40171 } } peers { permanent_uuid: "d150d8a6b6914c10b8aabc479a4e0067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 } } peers { permanent_uuid: "4af7dafec112490490274030f7f9ee57" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43665 } }
I20260924 03:14:16.299337 20760 ts_tablet_manager.cc:1428] T 635df502380b4efd976e53fd37106e56 P 4af7dafec112490490274030f7f9ee57: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260924 03:14:16.337205 20763 raft_consensus.cc:482] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:14:16.337586 20763 raft_consensus.cc:504] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d05b0b2c0841aabc56c5f38daf964f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40171 } } peers { permanent_uuid: "d150d8a6b6914c10b8aabc479a4e0067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 } } peers { permanent_uuid: "4af7dafec112490490274030f7f9ee57" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43665 } }
I20260924 03:14:16.339556 20763 leader_election.cc:288] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d150d8a6b6914c10b8aabc479a4e0067 (127.0.97.2:38219), 4af7dafec112490490274030f7f9ee57 (127.0.97.1:43665)
I20260924 03:14:16.346328 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.340000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50361 (local address 127.0.97.2:38219)
0924 03:14:16.340202 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:16.340207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:16.340296 (+ 89us) server_negotiation.cc:407] Connection header received
0924 03:14:16.340382 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:16.340387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:16.340534 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:16.340718 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:16.341278 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.342195 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.344130 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.345304 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.345406 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:16.345477 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:16.345893 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 03:14:16.346028 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 03:14:16.346382 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.339793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38219 (local address 127.0.97.3:50361)
0924 03:14:16.340014 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:16.340036 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:16.340197 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:16.340767 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:16.340772 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:16.340803 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:16.341020 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:16.341032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.342339 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.342345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:16.343955 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:16.343968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.345469 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.345495 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:16.345685 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:16.345753 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 03:14:16.345849 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 03:14:16.346033 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 03:14:16.346733 20765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.339988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43665 (local address 127.0.97.3:52387)
0924 03:14:16.340806 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:16.340819 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:16.340916 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:16.341307 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:16.341312 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:16.341342 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:14:16.341677 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:16.341689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.342899 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.342903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:16.344428 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:16.344442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.345855 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.345862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:16.346059 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:16.346090 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 03:14:16.346278 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 03:14:16.346426 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":518,"threads_started":1}
I20260924 03:14:16.347256 20755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.340208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52387 (local address 127.0.97.1:43665)
0924 03:14:16.340412 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:16.340418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:16.340913 (+ 495us) server_negotiation.cc:407] Connection header received
0924 03:14:16.341033 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:16.341038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:16.341135 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:16.341271 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:14:16.341845 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.342733 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.344664 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.345653 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.345777 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:16.345893 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:16.346322 (+ 429us) server_negotiation.cc:299] Negotiation successful
0924 03:14:16.346472 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 03:14:16.347647 20563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "635df502380b4efd976e53fd37106e56" candidate_uuid: "83d05b0b2c0841aabc56c5f38daf964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d150d8a6b6914c10b8aabc479a4e0067" is_pre_election: true
I20260924 03:14:16.348155 20432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "635df502380b4efd976e53fd37106e56" candidate_uuid: "83d05b0b2c0841aabc56c5f38daf964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af7dafec112490490274030f7f9ee57" is_pre_election: true
I20260924 03:14:16.348337 20563 raft_consensus.cc:2454] T 635df502380b4efd976e53fd37106e56 P d150d8a6b6914c10b8aabc479a4e0067 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83d05b0b2c0841aabc56c5f38daf964f in term 0.
I20260924 03:14:16.348755 20432 raft_consensus.cc:2454] T 635df502380b4efd976e53fd37106e56 P 4af7dafec112490490274030f7f9ee57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83d05b0b2c0841aabc56c5f38daf964f in term 0.
I20260924 03:14:16.349237 20628 leader_election.cc:302] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83d05b0b2c0841aabc56c5f38daf964f, d150d8a6b6914c10b8aabc479a4e0067; no voters:
I20260924 03:14:16.349748 20763 raft_consensus.cc:2781] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:14:16.349916 20763 raft_consensus.cc:482] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:14:16.350025 20763 raft_consensus.cc:3037] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:16.352253 20763 raft_consensus.cc:504] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d05b0b2c0841aabc56c5f38daf964f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40171 } } peers { permanent_uuid: "d150d8a6b6914c10b8aabc479a4e0067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 } } peers { permanent_uuid: "4af7dafec112490490274030f7f9ee57" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43665 } }
I20260924 03:14:16.353027 20763 leader_election.cc:288] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f [CANDIDATE]: Term 1 election: Requested vote from peers d150d8a6b6914c10b8aabc479a4e0067 (127.0.97.2:38219), 4af7dafec112490490274030f7f9ee57 (127.0.97.1:43665)
I20260924 03:14:16.353521 20563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "635df502380b4efd976e53fd37106e56" candidate_uuid: "83d05b0b2c0841aabc56c5f38daf964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d150d8a6b6914c10b8aabc479a4e0067"
I20260924 03:14:16.353595 20432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "635df502380b4efd976e53fd37106e56" candidate_uuid: "83d05b0b2c0841aabc56c5f38daf964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af7dafec112490490274030f7f9ee57"
I20260924 03:14:16.353740 20563 raft_consensus.cc:3037] T 635df502380b4efd976e53fd37106e56 P d150d8a6b6914c10b8aabc479a4e0067 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:16.353849 20432 raft_consensus.cc:3037] T 635df502380b4efd976e53fd37106e56 P 4af7dafec112490490274030f7f9ee57 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:16.356387 20563 raft_consensus.cc:2454] T 635df502380b4efd976e53fd37106e56 P d150d8a6b6914c10b8aabc479a4e0067 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83d05b0b2c0841aabc56c5f38daf964f in term 1.
I20260924 03:14:16.356750 20432 raft_consensus.cc:2454] T 635df502380b4efd976e53fd37106e56 P 4af7dafec112490490274030f7f9ee57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83d05b0b2c0841aabc56c5f38daf964f in term 1.
I20260924 03:14:16.356966 20628 leader_election.cc:302] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83d05b0b2c0841aabc56c5f38daf964f, d150d8a6b6914c10b8aabc479a4e0067; no voters:
I20260924 03:14:16.357367 20763 raft_consensus.cc:2781] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:16.357869 20763 raft_consensus.cc:686] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f [term 1 LEADER]: Becoming Leader. State: Replica: 83d05b0b2c0841aabc56c5f38daf964f, State: Running, Role: LEADER
I20260924 03:14:16.358364 20763 consensus_queue.cc:237] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d05b0b2c0841aabc56c5f38daf964f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40171 } } peers { permanent_uuid: "d150d8a6b6914c10b8aabc479a4e0067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 } } peers { permanent_uuid: "4af7dafec112490490274030f7f9ee57" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43665 } }
I20260924 03:14:16.363255 20290 catalog_manager.cc:5495] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f reported cstate change: term changed from 0 to 1, leader changed from <none> to 83d05b0b2c0841aabc56c5f38daf964f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "83d05b0b2c0841aabc56c5f38daf964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d05b0b2c0841aabc56c5f38daf964f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150d8a6b6914c10b8aabc479a4e0067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4af7dafec112490490274030f7f9ee57" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43665 } health_report { overall_health: UNKNOWN } } }
W20260924 03:14:16.397673 20739 tablet.cc:2369] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:14:16.397852 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.391589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.0.97.3:40171)
0924 03:14:16.392100 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:16.392104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:16.392118 (+ 14us) server_negotiation.cc:407] Connection header received
0924 03:14:16.392203 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:16.392207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:16.392297 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:16.392421 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:14:16.392961 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.393698 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.395168 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.395605 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.395841 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:16.396377 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:14:16.397090 (+ 713us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:14:16.397203 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:16.397365 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 03:14:16.397484 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396}
I20260924 03:14:16.406172 20432 raft_consensus.cc:1261] T 635df502380b4efd976e53fd37106e56 P 4af7dafec112490490274030f7f9ee57 [term 1 FOLLOWER]: Refusing update from remote peer 83d05b0b2c0841aabc56c5f38daf964f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:14:16.406170 20563 raft_consensus.cc:1261] T 635df502380b4efd976e53fd37106e56 P d150d8a6b6914c10b8aabc479a4e0067 [term 1 FOLLOWER]: Refusing update from remote peer 83d05b0b2c0841aabc56c5f38daf964f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:14:16.407132 20766 consensus_queue.cc:1035] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150d8a6b6914c10b8aabc479a4e0067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:14:16.407495 20763 consensus_queue.cc:1035] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af7dafec112490490274030f7f9ee57" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:14:16.421731 20770 mvcc.cc:205] Tried to move back new op lower bound from 7332739712628785152 to 7332739712448151552. Current Snapshot: MvccSnapshot[applied={T|T < 7332739712628785152}]
I20260924 03:14:16.426512 20771 mvcc.cc:205] Tried to move back new op lower bound from 7332739712628785152 to 7332739712448151552. Current Snapshot: MvccSnapshot[applied={T|T < 7332739712628785152}]
I20260924 03:14:16.437433 20291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:14:16.437770 20291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:14:16.441208 20291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e353c9e32ded4e5db2b3e88c510018d3: Sending DeleteTablet for 3 replicas of tablet 635df502380b4efd976e53fd37106e56
I20260924 03:14:16.442262 20673 tablet_service.cc:1514] Processing DeleteTablet for tablet 635df502380b4efd976e53fd37106e56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:14:16 UTC) from {username='slave'} at 127.0.0.1:40332
I20260924 03:14:16.442432 20543 tablet_service.cc:1514] Processing DeleteTablet for tablet 635df502380b4efd976e53fd37106e56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:14:16 UTC) from {username='slave'} at 127.0.0.1:55358
I20260924 03:14:16.443567 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 20350
I20260924 03:14:16.443508 20412 tablet_service.cc:1514] Processing DeleteTablet for tablet 635df502380b4efd976e53fd37106e56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:14:16 UTC) from {username='slave'} at 127.0.0.1:46752
I20260924 03:14:16.458123 20778 tablet_replica.cc:330] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f: stopping tablet replica
I20260924 03:14:16.458662 20778 raft_consensus.cc:2229] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:14:16.459409 20778 raft_consensus.cc:2258] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:14:16.462067 20778 ts_tablet_manager.cc:1905] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:14:16.466756 20755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.444918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.0.97.1:43665)
0924 03:14:16.445146 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:16.445155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:16.458137 (+ 12982us) server_negotiation.cc:407] Connection header received
0924 03:14:16.458653 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:16.458661 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:16.458795 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:16.458875 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:16.459556 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.460730 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.462558 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.463238 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.463350 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:16.464874 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:16.464886 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:16.464904 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:16.465683 (+ 779us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:16.465743 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:16.465770 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:16.465857 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:16.466130 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 03:14:16.466279 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 03:14:16.468633 20779 tablet_replica.cc:330] T 635df502380b4efd976e53fd37106e56 P d150d8a6b6914c10b8aabc479a4e0067: stopping tablet replica
I20260924 03:14:16.469167 20779 raft_consensus.cc:2229] T 635df502380b4efd976e53fd37106e56 P d150d8a6b6914c10b8aabc479a4e0067 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:14:16.469733 20779 raft_consensus.cc:2258] T 635df502380b4efd976e53fd37106e56 P d150d8a6b6914c10b8aabc479a4e0067 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:14:16.470058 20472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:16.470456 20778 ts_tablet_manager.cc:1918] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:14:16.470679 20778 log.cc:1196] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-2/wal/wals/635df502380b4efd976e53fd37106e56
I20260924 03:14:16.471514 20778 ts_tablet_manager.cc:1939] T 635df502380b4efd976e53fd37106e56 P 83d05b0b2c0841aabc56c5f38daf964f: Deleting consensus metadata
I20260924 03:14:16.474520 20779 ts_tablet_manager.cc:1905] T 635df502380b4efd976e53fd37106e56 P d150d8a6b6914c10b8aabc479a4e0067: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:14:16.475310 20275 catalog_manager.cc:4841] TS 83d05b0b2c0841aabc56c5f38daf964f (127.0.97.3:40171): tablet 635df502380b4efd976e53fd37106e56 (table test-table [id=ff0bb9e05ee143d6b90248cbc2ed3818]) successfully deleted
I20260924 03:14:16.482924 20779 ts_tablet_manager.cc:1918] T 635df502380b4efd976e53fd37106e56 P d150d8a6b6914c10b8aabc479a4e0067: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:14:16.483246 20779 log.cc:1196] T 635df502380b4efd976e53fd37106e56 P d150d8a6b6914c10b8aabc479a4e0067: Deleting WAL directory at /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219327163871-388-0/minicluster-data/ts-1/wal/wals/635df502380b4efd976e53fd37106e56
I20260924 03:14:16.483966 20779 ts_tablet_manager.cc:1939] T 635df502380b4efd976e53fd37106e56 P d150d8a6b6914c10b8aabc479a4e0067: Deleting consensus metadata
I20260924 03:14:16.486196 20277 catalog_manager.cc:4841] TS d150d8a6b6914c10b8aabc479a4e0067 (127.0.97.2:38219): tablet 635df502380b4efd976e53fd37106e56 (table test-table [id=ff0bb9e05ee143d6b90248cbc2ed3818]) successfully deleted
I20260924 03:14:16.565394 20780 tablet_replica.cc:330] T 635df502380b4efd976e53fd37106e56 P 4af7dafec112490490274030f7f9ee57: stopping tablet replica
I20260924 03:14:16.565618 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 20350
W20260924 03:14:16.584394 20276 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43665 (error 108)
I20260924 03:14:16.584609 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 20481
I20260924 03:14:16.587958 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.587353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43665 (local address 127.0.0.1:46782)
0924 03:14:16.587582 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:16.587752 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260924 03:14:16.588304 20276 catalog_manager.cc:4584] TS 4af7dafec112490490274030f7f9ee57 (127.0.97.1:43665): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 635df502380b4efd976e53fd37106e56: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43665: connect: Connection refused (error 111)
I20260924 03:14:16.593881 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.586071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55372 (local address 127.0.97.2:38219)
0924 03:14:16.586276 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:16.586282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:16.586301 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:14:16.586368 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:16.586375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:16.586478 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:16.586620 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:16.587148 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.588092 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.589978 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.590762 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.590924 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:16.592142 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:16.592151 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:16.592166 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:16.592913 (+ 747us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:16.592965 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:16.592985 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:16.593132 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:16.593327 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 03:14:16.593437 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:14:16.594751 20603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:16.642272 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.641675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43665 (local address 127.0.0.1:46784)
0924 03:14:16.641941 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:16.642052 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 03:14:16.643477 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.643021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43665 (local address 127.0.0.1:46788)
0924 03:14:16.643208 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:16.643307 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 03:14:16.697423 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 20481
I20260924 03:14:16.704450 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.703840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43665 (local address 127.0.0.1:46802)
0924 03:14:16.704108 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:16.704230 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 03:14:16.705703 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.705240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43665 (local address 127.0.0.1:46818)
0924 03:14:16.705434 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:16.705545 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 03:14:16.716337 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 20611
I20260924 03:14:16.723819 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.717368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40348 (local address 127.0.97.3:40171)
0924 03:14:16.717568 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:16.717574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:16.717610 (+ 36us) server_negotiation.cc:407] Connection header received
0924 03:14:16.717680 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:16.717686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:16.717774 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:16.717891 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:16.718301 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.719206 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.720629 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:16.721113 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:16.721286 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:16.722413 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:16.722423 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:16.722440 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:16.723024 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:16.723078 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:16.723098 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:16.723239 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:16.723355 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 03:14:16.723451 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:14:16.724651 20733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:16.786491 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.785991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43665 (local address 127.0.0.1:46830)
0924 03:14:16.786245 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:16.786338 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 03:14:16.787644 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:16.787244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43665 (local address 127.0.0.1:46846)
0924 03:14:16.787444 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:16.787537 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 03:14:16.845682 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 20611
I20260924 03:14:16.866372 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 20260
I20260924 03:14:16.867599 20321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:16.980285 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 20260
2026-09-24T03:14:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2936 ms)
[----------] 6 tests from EncryptionPolicyTest (22715 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:17 dist-test-slave-5l98 krb5kdc[20789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:17 dist-test-slave-5l98 krb5kdc[20789](info): set up 2 sockets
Sep 24 03:14:17 dist-test-slave-5l98 krb5kdc[20789](info): commencing operation
krb5kdc: starting...
W20260924 03:14:18.122941 388 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:18 dist-test-slave-5l98 krb5kdc[20789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:14:18.409783 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:18.749079 20802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:18.749374 20802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:18.749467 20802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:18.756706 20802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:18.756829 20802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:18.756877 20802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:18.756940 20802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:18.756985 20802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:18.767050 20802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35127
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:18.768348 20802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:18.769717 20802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:18.782191 20807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:18.782230 20810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:18.782207 20808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:18.786149 20802 server_base.cc:1034] running on GCE node
I20260924 03:14:18.787003 20802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:18.788586 20802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:18.789834 20802 hybrid_clock.cc:648] HybridClock initialized: now 1790219658789788 us; error 36 us; skew 500 ppm
I20260924 03:14:18.790190 20802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:18 dist-test-slave-5l98 krb5kdc[20789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:18.793433 20802 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:14:18.795291 20802 webserver.cc:460] Webserver started at http://127.0.0.1:40617/ using document root <none> and password file <none>
I20260924 03:14:18.796060 20802 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:18.796216 20802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:18.796609 20802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:18.799798 20802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "3568f6b5074b413895a0b89f9d49ef13"
format_stamp: "Formatted at 2026-09-24 03:14:18 on dist-test-slave-5l98"
server_key: "81c475dcab7b6255144159c2cde66e54"
server_key_iv: "999419f02d61b3c8d893ee1ead95fa19"
server_key_version: "encryptionkey@0"
I20260924 03:14:18.800701 20802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "3568f6b5074b413895a0b89f9d49ef13"
format_stamp: "Formatted at 2026-09-24 03:14:18 on dist-test-slave-5l98"
server_key: "81c475dcab7b6255144159c2cde66e54"
server_key_iv: "999419f02d61b3c8d893ee1ead95fa19"
server_key_version: "encryptionkey@0"
I20260924 03:14:18.806187 20802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 03:14:18.810030 20817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:18.811561 20802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:14:18.811781 20802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "3568f6b5074b413895a0b89f9d49ef13"
format_stamp: "Formatted at 2026-09-24 03:14:18 on dist-test-slave-5l98"
server_key: "81c475dcab7b6255144159c2cde66e54"
server_key_iv: "999419f02d61b3c8d893ee1ead95fa19"
server_key_version: "encryptionkey@0"
I20260924 03:14:18.812028 20802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:18.844172 20802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:18.849068 20802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:18.849421 20802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:18.869383 20802 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35127
I20260924 03:14:18.869405 20868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35127 every 8 connection(s)
I20260924 03:14:18.871119 20802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:18.875197 20869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:18.877534 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 20802
I20260924 03:14:18.877786 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:18.878594 388 external_mini_cluster.cc:1428] Setting key abee5ff68151487f3e6b73e8e7cc447e
I20260924 03:14:18.888669 20869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13: Bootstrap starting.
I20260924 03:14:18.892853 20869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:18.894310 20869 log.cc:824] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:14:18 dist-test-slave-5l98 krb5kdc[20789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:14:18.897511 20869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13: No bootstrap required, opened a new log
I20260924 03:14:18.903941 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:18.880901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54591 (local address 127.0.0.1:35127)
0924 03:14:18.881879 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:18.881892 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:18.881936 (+ 44us) server_negotiation.cc:407] Connection header received
0924 03:14:18.883710 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:18.883733 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:18.884144 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:18.884579 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:18.886451 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:18.887822 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:18.889712 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:18.890418 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:18.890562 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:18.895936 (+ 5374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:18.895945 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:18.895962 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:18.896000 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:18.899359 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:18.900293 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:18.900304 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:18.900311 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:18.900547 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:18.901078 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:18.901082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:18.901084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:18.901631 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:18.902046 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:18.902465 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 03:14:18.902824 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":501,"threads_started":1}
I20260924 03:14:18.905843 20869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568f6b5074b413895a0b89f9d49ef13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35127 } }
I20260924 03:14:18.906253 20869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:18.906385 20869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3568f6b5074b413895a0b89f9d49ef13, State: Initialized, Role: FOLLOWER
I20260924 03:14:18.906950 20869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568f6b5074b413895a0b89f9d49ef13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35127 } }
I20260924 03:14:18.907251 20869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:18.907423 20869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:18.907600 20869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:18.908409 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:14:18.910077 20869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568f6b5074b413895a0b89f9d49ef13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35127 } }
I20260924 03:14:18.910622 20869 leader_election.cc:302] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3568f6b5074b413895a0b89f9d49ef13; no voters:
I20260924 03:14:18.911361 20869 leader_election.cc:288] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:18.911569 20874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:18.913148 20874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13 [term 1 LEADER]: Becoming Leader. State: Replica: 3568f6b5074b413895a0b89f9d49ef13, State: Running, Role: LEADER
I20260924 03:14:18.913905 20874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568f6b5074b413895a0b89f9d49ef13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35127 } }
I20260924 03:14:18.914170 20869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:18.918162 20877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3568f6b5074b413895a0b89f9d49ef13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568f6b5074b413895a0b89f9d49ef13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35127 } } }
I20260924 03:14:18.918682 20877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:18.921041 20879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3568f6b5074b413895a0b89f9d49ef13. Latest consensus state: current_term: 1 leader_uuid: "3568f6b5074b413895a0b89f9d49ef13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568f6b5074b413895a0b89f9d49ef13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35127 } } }
I20260924 03:14:18.921540 20879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:18.925531 20886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:18.931546 20886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:18.940716 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:18.914953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41821 (local address 127.0.0.1:35127)
0924 03:14:18.915134 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:18.915140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:18.917007 (+ 1867us) server_negotiation.cc:407] Connection header received
0924 03:14:18.917317 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:18.917322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:18.917416 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:18.917497 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:18.919825 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:18.921512 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:18.924158 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:18.925166 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:18.925339 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:18.929511 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:18.929521 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:18.929541 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:18.929605 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:18.934862 (+ 5257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:18.936397 (+ 1535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:18.936406 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:18.936412 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:18.936641 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:18.937380 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:18.937388 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:18.937394 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:18.938030 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:18.938392 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:18.939904 (+ 1512us) server_negotiation.cc:299] Negotiation successful
0924 03:14:18.940108 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 03:14:18.944099 20886 catalog_manager.cc:1347] Generated new cluster ID: fb2993f706354edfbe1912204b699ff2
I20260924 03:14:18.944219 20886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:18.957859 20886 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:18.959192 20886 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:18.974328 20886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3568f6b5074b413895a0b89f9d49ef13: Generated new TSK 0
I20260924 03:14:18.975322 20886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:14:18.984118 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 20802
I20260924 03:14:18.986059 20864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:19.078984 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 20802
W20260924 03:14:19.093456 20787 connection.cc:535] client connection to 127.0.0.1:35127 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35127): Connection reset by peer (error 104)
2026-09-24T03:14:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2093 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:19 dist-test-slave-5l98 krb5kdc[20901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:19 dist-test-slave-5l98 krb5kdc[20901](info): set up 2 sockets
Sep 24 03:14:19 dist-test-slave-5l98 krb5kdc[20901](info): commencing operation
krb5kdc: starting...
W20260924 03:14:20.224994 388 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:20 dist-test-slave-5l98 krb5kdc[20901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:14:20.500186 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41205
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:20.840978 20914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:20.841324 20914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:20.841504 20914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:20.849265 20914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:20.849396 20914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:20.849498 20914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:20.849635 20914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:20.849713 20914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:20.860852 20914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41205
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:20.862264 20914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:20.863615 20914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:20.876227 20919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:20.876304 20920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:20.876255 20922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:20.877031 20914 server_base.cc:1034] running on GCE node
I20260924 03:14:20.877914 20914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:20.879489 20914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:20.880812 20914 hybrid_clock.cc:648] HybridClock initialized: now 1790219660880759 us; error 41 us; skew 500 ppm
I20260924 03:14:20.881224 20914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:20 dist-test-slave-5l98 krb5kdc[20901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:20.884662 20914 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:14:20.886462 20914 webserver.cc:460] Webserver started at http://127.0.0.1:35495/ using document root <none> and password file <none>
I20260924 03:14:20.887151 20914 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:20.887255 20914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:20.887513 20914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:20.890674 20914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "31df3596975e4d33b2140337bc7f6dea"
format_stamp: "Formatted at 2026-09-24 03:14:20 on dist-test-slave-5l98"
server_key: "9adb781cd0546b936c859e28e76b541f"
server_key_iv: "8a3c8083589309b2d84634649378de44"
server_key_version: "encryptionkey@0"
I20260924 03:14:20.891527 20914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "31df3596975e4d33b2140337bc7f6dea"
format_stamp: "Formatted at 2026-09-24 03:14:20 on dist-test-slave-5l98"
server_key: "9adb781cd0546b936c859e28e76b541f"
server_key_iv: "8a3c8083589309b2d84634649378de44"
server_key_version: "encryptionkey@0"
I20260924 03:14:20.898018 20914 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260924 03:14:20.902117 20929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:20.903614 20914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:14:20.903851 20914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "31df3596975e4d33b2140337bc7f6dea"
format_stamp: "Formatted at 2026-09-24 03:14:20 on dist-test-slave-5l98"
server_key: "9adb781cd0546b936c859e28e76b541f"
server_key_iv: "8a3c8083589309b2d84634649378de44"
server_key_version: "encryptionkey@0"
I20260924 03:14:20.904074 20914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:20.920966 20914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:20.925303 20914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:20.925592 20914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:20.945387 20914 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41205
I20260924 03:14:20.945430 20980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41205 every 8 connection(s)
I20260924 03:14:20.946892 20914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:20.950680 20981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:20.957070 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 20914
I20260924 03:14:20.957295 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:20.957764 388 external_mini_cluster.cc:1428] Setting key b0f15236fa7e41b946afb402cd417e35
I20260924 03:14:20.962138 20981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea: Bootstrap starting.
I20260924 03:14:20.965495 20981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:20.966547 20981 log.cc:824] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:20.969729 20981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea: No bootstrap required, opened a new log
I20260924 03:14:20.971246 20984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:20.959711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53109 (local address 127.0.0.1:41205)
0924 03:14:20.960591 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:20.960603 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:20.960644 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:14:20.962455 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:20.962479 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:20.962874 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:20.963269 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:20.963841 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:20.965218 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:20.966664 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:20.967421 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:20.967646 (+ 225us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:20.969021 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:20.969029 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:20.969044 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:20.969575 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:20.969674 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:20.969699 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:20.969879 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:20.970144 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 03:14:20.970325 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":358,"threads_started":1}
I20260924 03:14:20.975243 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:14:20.978147 20981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31df3596975e4d33b2140337bc7f6dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41205 } }
I20260924 03:14:20.978545 20981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:20.978665 20981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31df3596975e4d33b2140337bc7f6dea, State: Initialized, Role: FOLLOWER
I20260924 03:14:20.979219 20981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31df3596975e4d33b2140337bc7f6dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41205 } }
I20260924 03:14:20.979462 20981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:20.979610 20981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:20.979806 20981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:20.983004 20981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31df3596975e4d33b2140337bc7f6dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41205 } }
I20260924 03:14:20.983512 20981 leader_election.cc:302] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31df3596975e4d33b2140337bc7f6dea; no voters:
I20260924 03:14:20.984203 20981 leader_election.cc:288] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:20.984330 20991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:20.987008 20981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:20.987363 20984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:20.979807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53617 (local address 127.0.0.1:41205)
0924 03:14:20.980007 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:20.980013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:20.980471 (+ 458us) server_negotiation.cc:407] Connection header received
0924 03:14:20.980702 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:20.980707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:20.980790 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:20.980907 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:20.981380 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:20.982534 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:20.984070 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:20.985065 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:20.985187 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:20.986554 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:20.986559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:20.986562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:20.986580 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:20.986621 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:20.986635 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:20.986763 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:20.986942 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 03:14:20.987001 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:14:20.988802 20991 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea [term 1 LEADER]: Becoming Leader. State: Replica: 31df3596975e4d33b2140337bc7f6dea, State: Running, Role: LEADER
I20260924 03:14:20.989519 20991 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31df3596975e4d33b2140337bc7f6dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41205 } }
I20260924 03:14:20.993075 20992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31df3596975e4d33b2140337bc7f6dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31df3596975e4d33b2140337bc7f6dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41205 } } }
I20260924 03:14:20.993018 20993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31df3596975e4d33b2140337bc7f6dea. Latest consensus state: current_term: 1 leader_uuid: "31df3596975e4d33b2140337bc7f6dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31df3596975e4d33b2140337bc7f6dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41205 } } }
I20260924 03:14:20.993585 20992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:20.993592 20993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:20.995640 20999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:21.002071 20999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:21.013011 20999 catalog_manager.cc:1347] Generated new cluster ID: b8e42218fa4948ac8c6f987dc0d3aa85
I20260924 03:14:21.013152 20999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:21.042737 20999 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:21.044687 20999 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:21.060056 20999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31df3596975e4d33b2140337bc7f6dea: Generated new TSK 0
I20260924 03:14:21.061137 20999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:14:21.084010 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 20914
W20260924 03:14:21.085646 20976 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53109
W20260924 03:14:21.086510 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 20914: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:14:21.086792 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 20914
W20260924 03:14:21.103839 20899 connection.cc:535] client connection to 127.0.0.1:41205 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41205): Connection reset by peer (error 104)
2026-09-24T03:14:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2009 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:21 dist-test-slave-5l98 krb5kdc[21012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:21 dist-test-slave-5l98 krb5kdc[21012](info): set up 2 sockets
Sep 24 03:14:21 dist-test-slave-5l98 krb5kdc[21012](info): commencing operation
krb5kdc: starting...
W20260924 03:14:22.225405 388 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:22 dist-test-slave-5l98 krb5kdc[21012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:14:22.496471 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39993
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:22.830226 21025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:22.830533 21025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:22.830627 21025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:22.837916 21025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:22.838042 21025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:22.838088 21025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:22.838151 21025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:22.838194 21025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:22.848207 21025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39993
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:22.849581 21025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:22.851007 21025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:22.863904 21033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:22.863838 21031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:22.863848 21030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:22.864847 21025 server_base.cc:1034] running on GCE node
I20260924 03:14:22.865726 21025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:22.867369 21025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:22.868628 21025 hybrid_clock.cc:648] HybridClock initialized: now 1790219662868557 us; error 56 us; skew 500 ppm
I20260924 03:14:22.869095 21025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:22 dist-test-slave-5l98 krb5kdc[21012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:22.872859 21025 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:14:22.874688 21025 webserver.cc:460] Webserver started at http://127.0.0.1:33449/ using document root <none> and password file <none>
I20260924 03:14:22.875420 21025 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:22.875573 21025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:22.875900 21025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:22.879086 21025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "bb08d6146bec4d5b93123100069abaf0"
format_stamp: "Formatted at 2026-09-24 03:14:22 on dist-test-slave-5l98"
server_key: "68673865d8bc33fa10a3f3e053333769"
server_key_iv: "e7e02e82660f4ab6c2b9367628dd97a7"
server_key_version: "encryptionkey@0"
I20260924 03:14:22.879995 21025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "bb08d6146bec4d5b93123100069abaf0"
format_stamp: "Formatted at 2026-09-24 03:14:22 on dist-test-slave-5l98"
server_key: "68673865d8bc33fa10a3f3e053333769"
server_key_iv: "e7e02e82660f4ab6c2b9367628dd97a7"
server_key_version: "encryptionkey@0"
I20260924 03:14:22.885623 21025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:14:22.889356 21040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:22.890836 21025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:14:22.891052 21025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "bb08d6146bec4d5b93123100069abaf0"
format_stamp: "Formatted at 2026-09-24 03:14:22 on dist-test-slave-5l98"
server_key: "68673865d8bc33fa10a3f3e053333769"
server_key_iv: "e7e02e82660f4ab6c2b9367628dd97a7"
server_key_version: "encryptionkey@0"
I20260924 03:14:22.891322 21025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:22.897786 21025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:22.902443 21025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:22.902777 21025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:22.922902 21025 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39993
I20260924 03:14:22.922933 21091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39993 every 8 connection(s)
I20260924 03:14:22.924369 21025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:22.928174 21092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:22.932577 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 21025
I20260924 03:14:22.932775 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:22.933346 388 external_mini_cluster.cc:1428] Setting key 424d124ff29619d03a89d9ca79191d43
I20260924 03:14:22.941946 21095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:22.935209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37787 (local address 127.0.0.1:39993)
0924 03:14:22.936144 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:22.936157 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:22.936202 (+ 45us) server_negotiation.cc:407] Connection header received
0924 03:14:22.938077 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:22.938103 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:22.938525 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:22.938980 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:22.940239 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:22.940245 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:22.940259 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:22.940278 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:22.940331 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:22.940355 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:22.940628 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:22.940776 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 03:14:22.941001 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":424,"threads_started":1}
I20260924 03:14:22.941968 21092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0: Bootstrap starting.
I20260924 03:14:22.945324 21092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:22.945871 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:14:22.946468 21092 log.cc:824] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:22.949282 21092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0: No bootstrap required, opened a new log
I20260924 03:14:22.953512 21095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:22.950880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33883 (local address 127.0.0.1:39993)
0924 03:14:22.951121 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:22.951127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:22.951412 (+ 285us) server_negotiation.cc:407] Connection header received
0924 03:14:22.951618 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:22.951623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:22.951754 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:22.951854 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:22.952797 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:22.952801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:22.952804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:22.952829 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:22.952876 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:22.952895 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:22.952999 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:22.953133 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 03:14:22.953187 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 03:14:22.958348 21092 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb08d6146bec4d5b93123100069abaf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } }
I20260924 03:14:22.958765 21092 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:22.958896 21092 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb08d6146bec4d5b93123100069abaf0, State: Initialized, Role: FOLLOWER
I20260924 03:14:22.959507 21092 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb08d6146bec4d5b93123100069abaf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } }
I20260924 03:14:22.959753 21092 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:22.959903 21092 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:22.960073 21092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:22.962705 21092 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb08d6146bec4d5b93123100069abaf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } }
I20260924 03:14:22.963236 21092 leader_election.cc:302] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb08d6146bec4d5b93123100069abaf0; no voters:
I20260924 03:14:22.963855 21092 leader_election.cc:288] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:22.964051 21102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:22.965047 21102 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0 [term 1 LEADER]: Becoming Leader. State: Replica: bb08d6146bec4d5b93123100069abaf0, State: Running, Role: LEADER
I20260924 03:14:22.965587 21102 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb08d6146bec4d5b93123100069abaf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } }
I20260924 03:14:22.966250 21092 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:22.968564 21103 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb08d6146bec4d5b93123100069abaf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb08d6146bec4d5b93123100069abaf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } } }
I20260924 03:14:22.969094 21103 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:22.969686 21104 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb08d6146bec4d5b93123100069abaf0. Latest consensus state: current_term: 1 leader_uuid: "bb08d6146bec4d5b93123100069abaf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb08d6146bec4d5b93123100069abaf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } } }
I20260924 03:14:22.970057 21104 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:22.970163 21108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:22.982759 21108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:22.993957 21108 catalog_manager.cc:1347] Generated new cluster ID: 5353e20eb5424c149b5019a3f0490b14
I20260924 03:14:22.994081 21108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:23.010952 21108 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:23.011400 21108 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:23.024266 21108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb08d6146bec4d5b93123100069abaf0: Generated new TSK 0
I20260924 03:14:23.025290 21108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:14:23.036209 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 21025
W20260924 03:14:23.037357 21087 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37787
W20260924 03:14:23.038206 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 21025: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:14:23.038388 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 21025
2026-09-24T03:14:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1949 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:23 dist-test-slave-5l98 krb5kdc[21124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:23 dist-test-slave-5l98 krb5kdc[21124](info): set up 2 sockets
Sep 24 03:14:23 dist-test-slave-5l98 krb5kdc[21124](info): commencing operation
krb5kdc: starting...
W20260924 03:14:24.156906 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.008s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:24 dist-test-slave-5l98 krb5kdc[21124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:14:24.436992 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42913
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:24.784135 21137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:24.784508 21137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:24.784735 21137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:24.792459 21137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:24.792601 21137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:24.792704 21137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:24.792846 21137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:24.792906 21137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:24.803189 21137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:24.804610 21137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:24.806107 21137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:24.820791 21142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:24.820839 21143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:24.820883 21145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:24.822235 21137 server_base.cc:1034] running on GCE node
I20260924 03:14:24.823081 21137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:24.824674 21137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:24.825943 21137 hybrid_clock.cc:648] HybridClock initialized: now 1790219664825899 us; error 37 us; skew 500 ppm
I20260924 03:14:24.826345 21137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:24 dist-test-slave-5l98 krb5kdc[21124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:24.829897 21137 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:14:24.831585 21137 webserver.cc:460] Webserver started at http://127.0.0.1:39883/ using document root <none> and password file <none>
I20260924 03:14:24.832304 21137 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:24.832455 21137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:24.832824 21137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:24.836048 21137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "d2684655e0b5478bb36ee43d81bf0137"
format_stamp: "Formatted at 2026-09-24 03:14:24 on dist-test-slave-5l98"
server_key: "8a89d50905cbc39e1e259eb9ba0eb5c0"
server_key_iv: "19f3aa872685b910261e0f33ae49a307"
server_key_version: "encryptionkey@0"
I20260924 03:14:24.836951 21137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "d2684655e0b5478bb36ee43d81bf0137"
format_stamp: "Formatted at 2026-09-24 03:14:24 on dist-test-slave-5l98"
server_key: "8a89d50905cbc39e1e259eb9ba0eb5c0"
server_key_iv: "19f3aa872685b910261e0f33ae49a307"
server_key_version: "encryptionkey@0"
I20260924 03:14:24.842497 21137 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260924 03:14:24.846611 21152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:24.848313 21137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:14:24.848595 21137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "d2684655e0b5478bb36ee43d81bf0137"
format_stamp: "Formatted at 2026-09-24 03:14:24 on dist-test-slave-5l98"
server_key: "8a89d50905cbc39e1e259eb9ba0eb5c0"
server_key_iv: "19f3aa872685b910261e0f33ae49a307"
server_key_version: "encryptionkey@0"
I20260924 03:14:24.848870 21137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:24.862097 21137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:24.867398 21137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:24.867753 21137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:24.888413 21137 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42913
I20260924 03:14:24.888466 21203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42913 every 8 connection(s)
I20260924 03:14:24.890036 21137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:24.894057 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 21137
I20260924 03:14:24.894270 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:24.894713 388 external_mini_cluster.cc:1428] Setting key a0a3ff232fe1e9b4340fb49390249fea
I20260924 03:14:24.894846 21204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:24.911608 21204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137: Bootstrap starting.
Sep 24 03:14:24 dist-test-slave-5l98 krb5kdc[21124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:14:24.916303 21204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:24.917934 21204 log.cc:824] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:24.922286 21204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137: No bootstrap required, opened a new log
I20260924 03:14:24.923552 21207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:24.896817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54713 (local address 127.0.0.1:42913)
0924 03:14:24.897715 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:24.897731 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:24.897785 (+ 54us) server_negotiation.cc:407] Connection header received
0924 03:14:24.900544 (+ 2759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:24.900579 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:24.901270 (+ 691us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:24.901783 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:24.903282 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:24.905680 (+ 2398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:24.907228 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:24.908597 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:24.908820 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:24.913365 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:24.913384 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:24.913409 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:24.913490 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:24.918103 (+ 4613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:24.919240 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:24.919249 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:24.919256 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:24.919495 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:24.919956 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:24.919961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:24.919964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:24.920763 (+ 799us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:24.921110 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:24.921542 (+ 432us) server_negotiation.cc:299] Negotiation successful
0924 03:14:24.922021 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":373,"threads_started":1}
I20260924 03:14:24.927937 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:14:24.933930 21204 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2684655e0b5478bb36ee43d81bf0137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42913 } }
I20260924 03:14:24.934592 21204 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:24.934758 21204 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2684655e0b5478bb36ee43d81bf0137, State: Initialized, Role: FOLLOWER
I20260924 03:14:24.935508 21204 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2684655e0b5478bb36ee43d81bf0137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42913 } }
I20260924 03:14:24.935796 21204 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:24.935976 21204 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:24.936180 21204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:24.938930 21204 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2684655e0b5478bb36ee43d81bf0137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42913 } }
I20260924 03:14:24.939581 21204 leader_election.cc:302] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2684655e0b5478bb36ee43d81bf0137; no voters:
I20260924 03:14:24.940349 21204 leader_election.cc:288] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:24.940482 21214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:24.943427 21204 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:24.948575 21214 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137 [term 1 LEADER]: Becoming Leader. State: Replica: d2684655e0b5478bb36ee43d81bf0137, State: Running, Role: LEADER
I20260924 03:14:24.949394 21214 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2684655e0b5478bb36ee43d81bf0137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42913 } }
I20260924 03:14:24.950493 21207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:24.934072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49197 (local address 127.0.0.1:42913)
0924 03:14:24.934296 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:24.934303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:24.934767 (+ 464us) server_negotiation.cc:407] Connection header received
0924 03:14:24.935033 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:24.935040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:24.935129 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:24.935247 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:24.936969 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:24.937758 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:24.939071 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:24.939588 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:24.939736 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:24.942648 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:24.942653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:24.942656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:24.942689 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:24.946165 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:24.947260 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:24.947263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:24.947265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:24.947383 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:24.947800 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:24.947806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:24.947809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:24.948230 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:24.948609 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:24.950014 (+ 1405us) server_negotiation.cc:299] Negotiation successful
0924 03:14:24.950188 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 03:14:24.954327 21215 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2684655e0b5478bb36ee43d81bf0137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2684655e0b5478bb36ee43d81bf0137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42913 } } }
I20260924 03:14:24.954687 21215 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:24.955096 21216 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2684655e0b5478bb36ee43d81bf0137. Latest consensus state: current_term: 1 leader_uuid: "d2684655e0b5478bb36ee43d81bf0137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2684655e0b5478bb36ee43d81bf0137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42913 } } }
I20260924 03:14:24.955467 21216 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:24.957938 21228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:24.962692 21228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:24.971572 21228 catalog_manager.cc:1347] Generated new cluster ID: 47c08d3560a14f85a83b29c105f317db
I20260924 03:14:24.971705 21228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:24.994235 21228 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:24.995544 21228 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:25.013381 21228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2684655e0b5478bb36ee43d81bf0137: Generated new TSK 0
I20260924 03:14:25.014459 21228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:14:25.030725 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 21137
I20260924 03:14:25.031891 21199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:25.126555 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 21137
2026-09-24T03:14:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2092 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:25 dist-test-slave-5l98 krb5kdc[21236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:25 dist-test-slave-5l98 krb5kdc[21236](info): set up 2 sockets
Sep 24 03:14:25 dist-test-slave-5l98 krb5kdc[21236](info): commencing operation
krb5kdc: starting...
W20260924 03:14:26.253120 388 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:26 dist-test-slave-5l98 krb5kdc[21236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:14:26.524735 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40173
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:26.869910 21249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:26.870299 21249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:26.870468 21249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:26.878330 21249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:26.878520 21249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:26.878638 21249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:26.878762 21249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:26.878844 21249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:26.889433 21249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:26.890913 21249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:26.892325 21249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:26.905112 21254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:26.905102 21257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:26.905294 21255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:26.906352 21249 server_base.cc:1034] running on GCE node
I20260924 03:14:26.907291 21249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:26.908984 21249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:26.910228 21249 hybrid_clock.cc:648] HybridClock initialized: now 1790219666910189 us; error 30 us; skew 500 ppm
I20260924 03:14:26.910669 21249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:26 dist-test-slave-5l98 krb5kdc[21236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:26.921734 21249 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:14:26.923530 21249 webserver.cc:460] Webserver started at http://127.0.0.1:45615/ using document root <none> and password file <none>
I20260924 03:14:26.924285 21249 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:26.924440 21249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:26.924794 21249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:26.927809 21249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "afa29c29c9e74a26855949ef9e10b5d7"
format_stamp: "Formatted at 2026-09-24 03:14:26 on dist-test-slave-5l98"
server_key: "365694c60fa75d019826d335c1b18ac2"
server_key_iv: "0101b736290fedda341d3359fe0dee7b"
server_key_version: "encryptionkey@0"
I20260924 03:14:26.928678 21249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "afa29c29c9e74a26855949ef9e10b5d7"
format_stamp: "Formatted at 2026-09-24 03:14:26 on dist-test-slave-5l98"
server_key: "365694c60fa75d019826d335c1b18ac2"
server_key_iv: "0101b736290fedda341d3359fe0dee7b"
server_key_version: "encryptionkey@0"
I20260924 03:14:26.934309 21249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:14:26.938217 21264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:26.940176 21249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:14:26.940385 21249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "afa29c29c9e74a26855949ef9e10b5d7"
format_stamp: "Formatted at 2026-09-24 03:14:26 on dist-test-slave-5l98"
server_key: "365694c60fa75d019826d335c1b18ac2"
server_key_iv: "0101b736290fedda341d3359fe0dee7b"
server_key_version: "encryptionkey@0"
I20260924 03:14:26.940692 21249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:26.967954 21249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:26.973021 21249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:26.973345 21249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:26.992918 21249 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40173
I20260924 03:14:26.992955 21315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40173 every 8 connection(s)
I20260924 03:14:26.994359 21249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:26.998211 21316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:27.001582 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 21249
I20260924 03:14:27.001789 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:27.002344 388 external_mini_cluster.cc:1428] Setting key 1c7cbeec258d772bb20cf91feb9ba0e8
I20260924 03:14:27.010785 21316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7: Bootstrap starting.
I20260924 03:14:27.014119 21316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:27.015375 21316 log.cc:824] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:27.016330 21319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:27.004467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38637 (local address 127.0.0.1:40173)
0924 03:14:27.005246 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:27.005266 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:27.005309 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:14:27.007095 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:27.007118 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:27.007570 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:27.008006 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:27.008609 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:27.009939 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:27.011425 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:27.012080 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:27.012306 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:27.013526 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:27.013534 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:27.013549 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:27.014148 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:27.014234 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:27.014291 (+ 57us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:27.014413 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:27.014667 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 03:14:27.014951 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":323,"threads_started":1}
I20260924 03:14:27.018570 21316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7: No bootstrap required, opened a new log
I20260924 03:14:27.020033 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:14:27.027309 21316 raft_consensus.cc:348] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa29c29c9e74a26855949ef9e10b5d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40173 } }
I20260924 03:14:27.027812 21316 raft_consensus.cc:374] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:27.027925 21316 raft_consensus.cc:729] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afa29c29c9e74a26855949ef9e10b5d7, State: Initialized, Role: FOLLOWER
I20260924 03:14:27.028551 21316 consensus_queue.cc:260] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa29c29c9e74a26855949ef9e10b5d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40173 } }
I20260924 03:14:27.028841 21316 raft_consensus.cc:388] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:27.028983 21316 raft_consensus.cc:482] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:27.029202 21316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:27.031733 21319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:27.024592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45721 (local address 127.0.0.1:40173)
0924 03:14:27.024797 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:27.024804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:27.025232 (+ 428us) server_negotiation.cc:407] Connection header received
0924 03:14:27.025390 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:27.025397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:27.025500 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:27.025649 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:27.026227 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:27.027295 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:27.029093 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:27.029816 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:27.029960 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:27.030998 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:27.031003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:27.031008 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:27.031031 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:27.031078 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:27.031099 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:27.031201 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:27.031332 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 03:14:27.031420 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:14:27.032207 21316 raft_consensus.cc:504] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa29c29c9e74a26855949ef9e10b5d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40173 } }
I20260924 03:14:27.032791 21316 leader_election.cc:302] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afa29c29c9e74a26855949ef9e10b5d7; no voters:
I20260924 03:14:27.033486 21316 leader_election.cc:288] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:27.033600 21326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:27.034322 21326 raft_consensus.cc:686] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7 [term 1 LEADER]: Becoming Leader. State: Replica: afa29c29c9e74a26855949ef9e10b5d7, State: Running, Role: LEADER
I20260924 03:14:27.034964 21326 consensus_queue.cc:237] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa29c29c9e74a26855949ef9e10b5d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40173 } }
I20260924 03:14:27.035938 21316 sys_catalog.cc:564] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:27.038534 21328 sys_catalog.cc:455] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afa29c29c9e74a26855949ef9e10b5d7. Latest consensus state: current_term: 1 leader_uuid: "afa29c29c9e74a26855949ef9e10b5d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa29c29c9e74a26855949ef9e10b5d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40173 } } }
I20260924 03:14:27.038506 21327 sys_catalog.cc:455] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afa29c29c9e74a26855949ef9e10b5d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa29c29c9e74a26855949ef9e10b5d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40173 } } }
I20260924 03:14:27.039000 21328 sys_catalog.cc:458] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:27.039026 21327 sys_catalog.cc:458] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:27.040995 21331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:27.047596 21331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:27.057554 21331 catalog_manager.cc:1347] Generated new cluster ID: f8f466a77ae944e6bae46faeb95adf62
I20260924 03:14:27.057678 21331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:27.077265 21331 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:27.078706 21331 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:27.093340 21331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afa29c29c9e74a26855949ef9e10b5d7: Generated new TSK 0
I20260924 03:14:27.094254 21331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:14:27.101686 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 21249
W20260924 03:14:27.102931 21311 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38637
W20260924 03:14:27.103816 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 21249: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:14:27.103976 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 21249
2026-09-24T03:14:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1975 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:27 dist-test-slave-5l98 krb5kdc[21347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:27 dist-test-slave-5l98 krb5kdc[21347](info): set up 2 sockets
Sep 24 03:14:27 dist-test-slave-5l98 krb5kdc[21347](info): commencing operation
krb5kdc: starting...
W20260924 03:14:28.333429 388 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:28 dist-test-slave-5l98 krb5kdc[21347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:14:28.612233 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39329
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:28.955866 21360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:28.956187 21360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:28.956279 21360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:28.963878 21360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:28.964010 21360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:28.964058 21360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:28.964111 21360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:28.964155 21360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:28.974426 21360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:28.975689 21360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:28.976954 21360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:28.989521 21366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:28.989494 21368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:28.989687 21365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:28.990617 21360 server_base.cc:1034] running on GCE node
I20260924 03:14:28.991518 21360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:28.993065 21360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:28.994274 21360 hybrid_clock.cc:648] HybridClock initialized: now 1790219668994230 us; error 36 us; skew 500 ppm
I20260924 03:14:28.994650 21360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:28 dist-test-slave-5l98 krb5kdc[21347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:28.998116 21360 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:14:28.999738 21360 webserver.cc:460] Webserver started at http://127.0.0.1:44387/ using document root <none> and password file <none>
I20260924 03:14:29.000418 21360 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:29.000578 21360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:29.000856 21360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:29.003890 21360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "d102cc59b5344aaeb395089733c7e229"
format_stamp: "Formatted at 2026-09-24 03:14:29 on dist-test-slave-5l98"
server_key: "59d33c7993ac366e3f4253f246749a0d"
server_key_iv: "d0c9e3781f2ea000e843308fc86a1200"
server_key_version: "encryptionkey@0"
I20260924 03:14:29.004721 21360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "d102cc59b5344aaeb395089733c7e229"
format_stamp: "Formatted at 2026-09-24 03:14:29 on dist-test-slave-5l98"
server_key: "59d33c7993ac366e3f4253f246749a0d"
server_key_iv: "d0c9e3781f2ea000e843308fc86a1200"
server_key_version: "encryptionkey@0"
I20260924 03:14:29.010357 21360 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:14:29.014058 21375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:29.015519 21360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 03:14:29.015815 21360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "d102cc59b5344aaeb395089733c7e229"
format_stamp: "Formatted at 2026-09-24 03:14:29 on dist-test-slave-5l98"
server_key: "59d33c7993ac366e3f4253f246749a0d"
server_key_iv: "d0c9e3781f2ea000e843308fc86a1200"
server_key_version: "encryptionkey@0"
I20260924 03:14:29.016039 21360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:29.022104 21360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:29.026311 21360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:29.026636 21360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:29.045872 21360 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39329
I20260924 03:14:29.045975 21426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39329 every 8 connection(s)
I20260924 03:14:29.047333 21360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:29.048775 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 21360
I20260924 03:14:29.048982 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:29.049427 388 external_mini_cluster.cc:1428] Setting key 73f91653b9861c44156879d86c5eb027
I20260924 03:14:29.052047 21427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:29.058583 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:29.051303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60899 (local address 127.0.0.1:39329)
0924 03:14:29.052170 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:29.052185 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:29.052233 (+ 48us) server_negotiation.cc:407] Connection header received
0924 03:14:29.054631 (+ 2398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:29.054658 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:29.055102 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:29.055558 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:29.056599 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:29.056607 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:29.056624 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:29.056642 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:29.056737 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:29.056761 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:29.056975 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:29.057112 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 03:14:29.057323 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":341,"threads_started":1}
I20260924 03:14:29.062808 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:14:29.065582 21427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229: Bootstrap starting.
I20260924 03:14:29.069415 21427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:29.070643 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:29.067667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40555 (local address 127.0.0.1:39329)
0924 03:14:29.067965 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:29.067972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:29.068271 (+ 299us) server_negotiation.cc:407] Connection header received
0924 03:14:29.068568 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:29.068572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:29.068633 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:29.068756 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:29.069890 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:29.069894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:29.069901 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:29.069918 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:29.069964 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:29.069980 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:29.070117 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:29.070273 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 03:14:29.070348 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:14:29.070677 21427 log.cc:824] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:29.073827 21427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229: No bootstrap required, opened a new log
I20260924 03:14:29.081983 21427 raft_consensus.cc:348] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d102cc59b5344aaeb395089733c7e229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39329 } }
I20260924 03:14:29.082594 21427 raft_consensus.cc:374] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:29.082719 21427 raft_consensus.cc:729] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d102cc59b5344aaeb395089733c7e229, State: Initialized, Role: FOLLOWER
I20260924 03:14:29.083436 21427 consensus_queue.cc:260] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d102cc59b5344aaeb395089733c7e229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39329 } }
I20260924 03:14:29.083691 21427 raft_consensus.cc:388] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:29.083847 21427 raft_consensus.cc:482] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:29.084028 21427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:29.086355 21427 raft_consensus.cc:504] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d102cc59b5344aaeb395089733c7e229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39329 } }
I20260924 03:14:29.086894 21427 leader_election.cc:302] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d102cc59b5344aaeb395089733c7e229; no voters:
I20260924 03:14:29.087585 21427 leader_election.cc:288] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:29.087733 21437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:29.088523 21437 raft_consensus.cc:686] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229 [term 1 LEADER]: Becoming Leader. State: Replica: d102cc59b5344aaeb395089733c7e229, State: Running, Role: LEADER
I20260924 03:14:29.089150 21437 consensus_queue.cc:237] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d102cc59b5344aaeb395089733c7e229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39329 } }
I20260924 03:14:29.090165 21427 sys_catalog.cc:564] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:29.092542 21439 sys_catalog.cc:455] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d102cc59b5344aaeb395089733c7e229. Latest consensus state: current_term: 1 leader_uuid: "d102cc59b5344aaeb395089733c7e229" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d102cc59b5344aaeb395089733c7e229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39329 } } }
I20260924 03:14:29.092577 21438 sys_catalog.cc:455] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d102cc59b5344aaeb395089733c7e229" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d102cc59b5344aaeb395089733c7e229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39329 } } }
I20260924 03:14:29.093129 21439 sys_catalog.cc:458] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:29.093175 21438 sys_catalog.cc:458] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:29.096375 21443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:29.102593 21443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:29.114241 21443 catalog_manager.cc:1347] Generated new cluster ID: cccc8579b3bc44e9843e8cba8dbd8c89
I20260924 03:14:29.114413 21443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:29.128036 21443 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:29.128584 21443 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:29.141266 21443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d102cc59b5344aaeb395089733c7e229: Generated new TSK 0
I20260924 03:14:29.142210 21443 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:14:29.157686 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 21360
W20260924 03:14:29.159077 21422 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60899
W20260924 03:14:29.159904 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 21360: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:14:29.160048 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 21360
2026-09-24T03:14:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2055 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:29 dist-test-slave-5l98 krb5kdc[21458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:29 dist-test-slave-5l98 krb5kdc[21458](info): set up 2 sockets
Sep 24 03:14:29 dist-test-slave-5l98 krb5kdc[21458](info): commencing operation
krb5kdc: starting...
W20260924 03:14:30.328984 388 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:30 dist-test-slave-5l98 krb5kdc[21458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:14:30.604655 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38367
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:30.947863 21471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:30.948231 21471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:30.948408 21471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:30.956085 21471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:30.956272 21471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:30.956394 21471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:30.956532 21471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:30.956615 21471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:30.966944 21471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:30.968425 21471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:30.969913 21471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:30.982295 21477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:30.982241 21479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:30.982542 21476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:30.983246 21471 server_base.cc:1034] running on GCE node
I20260924 03:14:30.984061 21471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:30.985635 21471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:30.986866 21471 hybrid_clock.cc:648] HybridClock initialized: now 1790219670986818 us; error 48 us; skew 500 ppm
I20260924 03:14:30.987218 21471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:30 dist-test-slave-5l98 krb5kdc[21458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:30.990608 21471 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:14:30.992266 21471 webserver.cc:460] Webserver started at http://127.0.0.1:35101/ using document root <none> and password file <none>
I20260924 03:14:30.993047 21471 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:30.993201 21471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:30.993507 21471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:30.996646 21471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "f86984d69285406fb6fe6418d90cc0ca"
format_stamp: "Formatted at 2026-09-24 03:14:30 on dist-test-slave-5l98"
server_key: "7ae514b22d00ff404ba11e65602eff7e"
server_key_iv: "f24c81e3538894e78b8332cb3de32bd8"
server_key_version: "encryptionkey@0"
I20260924 03:14:30.997463 21471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "f86984d69285406fb6fe6418d90cc0ca"
format_stamp: "Formatted at 2026-09-24 03:14:30 on dist-test-slave-5l98"
server_key: "7ae514b22d00ff404ba11e65602eff7e"
server_key_iv: "f24c81e3538894e78b8332cb3de32bd8"
server_key_version: "encryptionkey@0"
I20260924 03:14:31.002843 21471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:14:31.006655 21486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:31.008138 21471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:14:31.008460 21471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "f86984d69285406fb6fe6418d90cc0ca"
format_stamp: "Formatted at 2026-09-24 03:14:30 on dist-test-slave-5l98"
server_key: "7ae514b22d00ff404ba11e65602eff7e"
server_key_iv: "f24c81e3538894e78b8332cb3de32bd8"
server_key_version: "encryptionkey@0"
I20260924 03:14:31.008759 21471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:31.051261 21471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:31.055814 21471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:31.056114 21471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:31.075858 21471 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38367
I20260924 03:14:31.075947 21537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38367 every 8 connection(s)
I20260924 03:14:31.077239 21471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:31.081151 21538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:31.081741 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 21471
I20260924 03:14:31.081925 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:31.082489 388 external_mini_cluster.cc:1428] Setting key 50cf3e98072ad56a618b344f4a04d554
I20260924 03:14:31.094841 21538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca: Bootstrap starting.
Sep 24 03:14:31 dist-test-slave-5l98 krb5kdc[21458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:14:31.098722 21538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:31.099836 21538 log.cc:824] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:31.103262 21538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca: No bootstrap required, opened a new log
I20260924 03:14:31.106050 21541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:31.084275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:52941 (local address 127.0.0.1:38367)
0924 03:14:31.085110 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:31.085122 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:31.085165 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:14:31.086864 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:31.086889 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:31.087332 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:31.087787 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:31.089375 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:31.090678 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:31.092018 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:31.092844 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:31.093080 (+ 236us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:31.097994 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:31.098002 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:31.098018 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:31.098063 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:31.101658 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:31.102649 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:31.102654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:31.102660 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:31.102826 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:31.103385 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:31.103392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:31.103400 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:31.103890 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:31.104116 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:31.104685 (+ 569us) server_negotiation.cc:299] Negotiation successful
0924 03:14:31.105018 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":379,"threads_started":1}
I20260924 03:14:31.109889 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:14:31.111896 21538 raft_consensus.cc:348] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86984d69285406fb6fe6418d90cc0ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38367 } }
I20260924 03:14:31.112303 21538 raft_consensus.cc:374] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:31.112427 21538 raft_consensus.cc:729] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86984d69285406fb6fe6418d90cc0ca, State: Initialized, Role: FOLLOWER
I20260924 03:14:31.113149 21538 consensus_queue.cc:260] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86984d69285406fb6fe6418d90cc0ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38367 } }
I20260924 03:14:31.113406 21538 raft_consensus.cc:388] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:31.113560 21538 raft_consensus.cc:482] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:31.113749 21538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:31.116458 21538 raft_consensus.cc:504] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86984d69285406fb6fe6418d90cc0ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38367 } }
I20260924 03:14:31.117085 21538 leader_election.cc:302] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f86984d69285406fb6fe6418d90cc0ca; no voters:
I20260924 03:14:31.117786 21538 leader_election.cc:288] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:31.117913 21548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:31.119395 21548 raft_consensus.cc:686] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca [term 1 LEADER]: Becoming Leader. State: Replica: f86984d69285406fb6fe6418d90cc0ca, State: Running, Role: LEADER
I20260924 03:14:31.119961 21548 consensus_queue.cc:237] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86984d69285406fb6fe6418d90cc0ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38367 } }
I20260924 03:14:31.120162 21538 sys_catalog.cc:564] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:31.122759 21549 sys_catalog.cc:455] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f86984d69285406fb6fe6418d90cc0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86984d69285406fb6fe6418d90cc0ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38367 } } }
I20260924 03:14:31.123157 21549 sys_catalog.cc:458] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:31.127365 21550 sys_catalog.cc:455] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader f86984d69285406fb6fe6418d90cc0ca. Latest consensus state: current_term: 1 leader_uuid: "f86984d69285406fb6fe6418d90cc0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86984d69285406fb6fe6418d90cc0ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38367 } } }
I20260924 03:14:31.128192 21550 sys_catalog.cc:458] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:31.129817 21553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:31.135214 21553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:31.136247 21541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:31.114479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:55327 (local address 127.0.0.1:38367)
0924 03:14:31.114725 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:31.114734 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:31.115261 (+ 527us) server_negotiation.cc:407] Connection header received
0924 03:14:31.115380 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:31.115386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:31.115474 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:31.115584 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:31.117162 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:31.118485 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:31.120329 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:31.121020 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:31.121149 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:31.125062 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:31.125072 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:31.125080 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:31.125135 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:31.130155 (+ 5020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:31.131693 (+ 1538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:31.131702 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:31.131708 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:31.131909 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:31.132747 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:31.132752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:31.132754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:31.133376 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:31.135165 (+ 1789us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:31.135417 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 03:14:31.135678 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 03:14:31.146992 21553 catalog_manager.cc:1347] Generated new cluster ID: b729dc0e8a7947f3afd00b91d621c81c
I20260924 03:14:31.147171 21553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:31.165606 21553 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:31.167698 21553 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:31.180816 21553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f86984d69285406fb6fe6418d90cc0ca: Generated new TSK 0
I20260924 03:14:31.181897 21553 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:14:31.191694 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 21471
I20260924 03:14:31.194057 21533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:31.305207 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 21471
W20260924 03:14:31.323119 21456 connection.cc:535] client connection to 127.0.0.1:38367 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38367): Connection reset by peer (error 104)
2026-09-24T03:14:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2145 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:31 dist-test-slave-5l98 krb5kdc[21570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:31 dist-test-slave-5l98 krb5kdc[21570](info): set up 2 sockets
Sep 24 03:14:31 dist-test-slave-5l98 krb5kdc[21570](info): commencing operation
krb5kdc: starting...
W20260924 03:14:32.436977 388 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.008s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:32 dist-test-slave-5l98 krb5kdc[21570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:14:32.717404 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46791
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:33.061702 21583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:33.062072 21583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:33.062250 21583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:33.069756 21583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:33.069941 21583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:33.070060 21583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:33.070173 21583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:33.070251 21583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:33.080736 21583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:33.082121 21583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:33.083518 21583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:33.096369 21588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:33.096450 21589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:33.096401 21591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:33.097395 21583 server_base.cc:1034] running on GCE node
I20260924 03:14:33.098241 21583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:33.099788 21583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:33.101032 21583 hybrid_clock.cc:648] HybridClock initialized: now 1790219673100976 us; error 53 us; skew 500 ppm
I20260924 03:14:33.101389 21583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:33 dist-test-slave-5l98 krb5kdc[21570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:33.111184 21583 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:14:33.113050 21583 webserver.cc:460] Webserver started at http://127.0.0.1:37277/ using document root <none> and password file <none>
I20260924 03:14:33.113735 21583 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:33.113842 21583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:33.114094 21583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:33.117158 21583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "56e018de0cc940a7b9758cf70337056c"
format_stamp: "Formatted at 2026-09-24 03:14:33 on dist-test-slave-5l98"
server_key: "a7d26f1472b55474ec26d3602196b35e"
server_key_iv: "d373b8353f1358ecf5379d454a87c3e8"
server_key_version: "encryptionkey@0"
I20260924 03:14:33.117928 21583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "56e018de0cc940a7b9758cf70337056c"
format_stamp: "Formatted at 2026-09-24 03:14:33 on dist-test-slave-5l98"
server_key: "a7d26f1472b55474ec26d3602196b35e"
server_key_iv: "d373b8353f1358ecf5379d454a87c3e8"
server_key_version: "encryptionkey@0"
I20260924 03:14:33.123199 21583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:14:33.126946 21598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:33.128392 21583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 03:14:33.128631 21583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "56e018de0cc940a7b9758cf70337056c"
format_stamp: "Formatted at 2026-09-24 03:14:33 on dist-test-slave-5l98"
server_key: "a7d26f1472b55474ec26d3602196b35e"
server_key_iv: "d373b8353f1358ecf5379d454a87c3e8"
server_key_version: "encryptionkey@0"
I20260924 03:14:33.128823 21583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:33.143605 21583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:33.147878 21583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:33.148123 21583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:33.167995 21583 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46791
I20260924 03:14:33.168026 21649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46791 every 8 connection(s)
I20260924 03:14:33.169689 21583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:33.174033 21650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:33.174923 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 21583
I20260924 03:14:33.175155 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:33.175699 388 external_mini_cluster.cc:1428] Setting key 8df8453e589f7e5ec60cf94a0bbc9974
I20260924 03:14:33.188848 21650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c: Bootstrap starting.
I20260924 03:14:33.190508 21653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:33.177588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:50467 (local address 127.0.0.1:46791)
0924 03:14:33.178526 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:33.178540 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:33.178581 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:14:33.180856 (+ 2275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:33.180881 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:33.181333 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:33.181702 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:33.182349 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:33.183604 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:33.185657 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:33.186428 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:33.186561 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:33.187897 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:33.187904 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:33.187918 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:33.188550 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:33.188649 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:33.188674 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:33.188797 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:33.189282 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 03:14:33.189475 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":440,"threads_started":1}
I20260924 03:14:33.192225 21650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:33.193415 21650 log.cc:824] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:33.194867 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:14:33.196697 21650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c: No bootstrap required, opened a new log
I20260924 03:14:33.207072 21653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:33.199518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:54109 (local address 127.0.0.1:46791)
0924 03:14:33.199704 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:33.199711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:33.200267 (+ 556us) server_negotiation.cc:407] Connection header received
0924 03:14:33.200478 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:33.200482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:33.200622 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:33.200704 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:33.201348 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:33.202145 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:33.204254 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:33.204860 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:33.204970 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:33.206221 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:33.206225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:33.206229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:33.206250 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:33.206289 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:33.206304 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:33.206386 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:33.206602 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 03:14:33.206648 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 03:14:33.206851 21650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e018de0cc940a7b9758cf70337056c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46791 } }
I20260924 03:14:33.207302 21650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:33.207482 21650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e018de0cc940a7b9758cf70337056c, State: Initialized, Role: FOLLOWER
I20260924 03:14:33.208231 21650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e018de0cc940a7b9758cf70337056c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46791 } }
I20260924 03:14:33.208564 21650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:33.208724 21650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:33.208926 21650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:33.211241 21650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e018de0cc940a7b9758cf70337056c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46791 } }
I20260924 03:14:33.211829 21650 leader_election.cc:302] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56e018de0cc940a7b9758cf70337056c; no voters:
I20260924 03:14:33.212574 21650 leader_election.cc:288] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:33.212698 21660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:33.213461 21660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c [term 1 LEADER]: Becoming Leader. State: Replica: 56e018de0cc940a7b9758cf70337056c, State: Running, Role: LEADER
I20260924 03:14:33.214021 21660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e018de0cc940a7b9758cf70337056c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46791 } }
I20260924 03:14:33.214726 21650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:33.217093 21662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56e018de0cc940a7b9758cf70337056c. Latest consensus state: current_term: 1 leader_uuid: "56e018de0cc940a7b9758cf70337056c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e018de0cc940a7b9758cf70337056c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46791 } } }
I20260924 03:14:33.217662 21662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:33.218590 21661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56e018de0cc940a7b9758cf70337056c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e018de0cc940a7b9758cf70337056c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46791 } } }
I20260924 03:14:33.218968 21661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:33.220564 21668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:33.225528 21668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:33.234989 21668 catalog_manager.cc:1347] Generated new cluster ID: 629e0eb81fc543f5a69458811288f6c1
I20260924 03:14:33.235121 21668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:33.265980 21668 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:33.268010 21668 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:33.283607 21668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56e018de0cc940a7b9758cf70337056c: Generated new TSK 0
I20260924 03:14:33.284806 21668 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:14:33.294744 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 21583
W20260924 03:14:33.295918 21645 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.89:50467
W20260924 03:14:33.296720 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 21583: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:14:33.296883 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 21583
W20260924 03:14:33.317180 21568 connection.cc:535] client connection to 127.0.0.1:46791 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46791): Connection reset by peer (error 104)
2026-09-24T03:14:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1994 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:33 dist-test-slave-5l98 krb5kdc[21681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:33 dist-test-slave-5l98 krb5kdc[21681](info): set up 2 sockets
Sep 24 03:14:33 dist-test-slave-5l98 krb5kdc[21681](info): commencing operation
krb5kdc: starting...
W20260924 03:14:34.433362 388 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:34 dist-test-slave-5l98 krb5kdc[21681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:14:34.716435 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38727
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:35.057266 21694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:35.057621 21694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:35.057791 21694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:35.065493 21694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:35.065658 21694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:35.065775 21694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:35.065879 21694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:35.065961 21694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:35.076045 21694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:35.077440 21694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:35.078747 21694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:35.090583 21699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:35.090559 21700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:35.090636 21702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:35.091634 21694 server_base.cc:1034] running on GCE node
I20260924 03:14:35.092464 21694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:35.093981 21694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:35.095230 21694 hybrid_clock.cc:648] HybridClock initialized: now 1790219675095195 us; error 32 us; skew 500 ppm
I20260924 03:14:35.095592 21694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:35 dist-test-slave-5l98 krb5kdc[21681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:35.098987 21694 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:14:35.100615 21694 webserver.cc:460] Webserver started at http://127.0.0.1:33811/ using document root <none> and password file <none>
I20260924 03:14:35.101331 21694 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:35.101518 21694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:35.101857 21694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:35.105031 21694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "7321b744f73e4e1c9fafcb37cba2b08f"
format_stamp: "Formatted at 2026-09-24 03:14:35 on dist-test-slave-5l98"
server_key: "97dd31adca977db75b925cc2e08143ba"
server_key_iv: "e764df5319da40722e524237b362ee72"
server_key_version: "encryptionkey@0"
I20260924 03:14:35.105865 21694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "7321b744f73e4e1c9fafcb37cba2b08f"
format_stamp: "Formatted at 2026-09-24 03:14:35 on dist-test-slave-5l98"
server_key: "97dd31adca977db75b925cc2e08143ba"
server_key_iv: "e764df5319da40722e524237b362ee72"
server_key_version: "encryptionkey@0"
I20260924 03:14:35.111259 21694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:14:35.114964 21709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:35.116341 21694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 03:14:35.116577 21694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "7321b744f73e4e1c9fafcb37cba2b08f"
format_stamp: "Formatted at 2026-09-24 03:14:35 on dist-test-slave-5l98"
server_key: "97dd31adca977db75b925cc2e08143ba"
server_key_iv: "e764df5319da40722e524237b362ee72"
server_key_version: "encryptionkey@0"
I20260924 03:14:35.116789 21694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:35.122674 21694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:35.126855 21694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:35.127168 21694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:35.146435 21694 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38727
I20260924 03:14:35.146487 21760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38727 every 8 connection(s)
I20260924 03:14:35.147914 21694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:35.151810 21761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:35.153048 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 21694
I20260924 03:14:35.153249 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:35.153813 388 external_mini_cluster.cc:1428] Setting key bdf71b87e0bd579d71b876e8caab6990
I20260924 03:14:35.162241 21764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:35.155671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:39235 (local address 127.0.0.1:38727)
0924 03:14:35.156606 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:35.156617 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:35.156659 (+ 42us) server_negotiation.cc:407] Connection header received
0924 03:14:35.158546 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:35.158572 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:35.158961 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:35.159385 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:35.160483 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:35.160538 (+ 55us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:35.160557 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:35.160575 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:35.160645 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:35.160667 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:35.160909 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:35.161044 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 03:14:35.161233 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":347,"threads_started":1}
I20260924 03:14:35.166266 21761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f: Bootstrap starting.
I20260924 03:14:35.166313 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:14:35.169574 21761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:35.170802 21761 log.cc:824] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:35.174036 21764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:35.171072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:47963 (local address 127.0.0.1:38727)
0924 03:14:35.171293 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:35.171300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:35.171648 (+ 348us) server_negotiation.cc:407] Connection header received
0924 03:14:35.171866 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:35.171873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:35.171948 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:35.172072 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:35.173246 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:35.173273 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:35.173277 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:35.173294 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:35.173342 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:35.173359 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:35.173461 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:35.173636 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 03:14:35.173693 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 03:14:35.174867 21761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f: No bootstrap required, opened a new log
I20260924 03:14:35.182775 21761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7321b744f73e4e1c9fafcb37cba2b08f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38727 } }
I20260924 03:14:35.183151 21761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:35.183238 21761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7321b744f73e4e1c9fafcb37cba2b08f, State: Initialized, Role: FOLLOWER
I20260924 03:14:35.183748 21761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7321b744f73e4e1c9fafcb37cba2b08f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38727 } }
I20260924 03:14:35.183962 21761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:35.184062 21761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:35.184182 21761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:35.186455 21761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7321b744f73e4e1c9fafcb37cba2b08f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38727 } }
I20260924 03:14:35.186966 21761 leader_election.cc:302] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7321b744f73e4e1c9fafcb37cba2b08f; no voters:
I20260924 03:14:35.187512 21761 leader_election.cc:288] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:35.187686 21771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:35.188423 21771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f [term 1 LEADER]: Becoming Leader. State: Replica: 7321b744f73e4e1c9fafcb37cba2b08f, State: Running, Role: LEADER
I20260924 03:14:35.189061 21771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7321b744f73e4e1c9fafcb37cba2b08f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38727 } }
I20260924 03:14:35.189782 21761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:35.191761 21772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7321b744f73e4e1c9fafcb37cba2b08f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7321b744f73e4e1c9fafcb37cba2b08f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38727 } } }
I20260924 03:14:35.192260 21772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:35.192737 21773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7321b744f73e4e1c9fafcb37cba2b08f. Latest consensus state: current_term: 1 leader_uuid: "7321b744f73e4e1c9fafcb37cba2b08f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7321b744f73e4e1c9fafcb37cba2b08f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38727 } } }
I20260924 03:14:35.193198 21773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:35.195484 21776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:35.201457 21776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:35.211139 21776 catalog_manager.cc:1347] Generated new cluster ID: 61c826d64c3b4755a1e2d127c04d0a69
I20260924 03:14:35.211274 21776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:35.231705 21776 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:35.232213 21776 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:35.250293 21776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7321b744f73e4e1c9fafcb37cba2b08f: Generated new TSK 0
I20260924 03:14:35.251516 21776 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:14:35.262142 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 21694
W20260924 03:14:35.263393 21756 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.89:39235
W20260924 03:14:35.264189 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 21694: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:14:35.264328 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 21694
2026-09-24T03:14:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1964 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:35 dist-test-slave-5l98 krb5kdc[21792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:35 dist-test-slave-5l98 krb5kdc[21792](info): set up 2 sockets
Sep 24 03:14:35 dist-test-slave-5l98 krb5kdc[21792](info): commencing operation
krb5kdc: starting...
W20260924 03:14:36.401140 388 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:36 dist-test-slave-5l98 krb5kdc[21792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:14:36.673103 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44287
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:37.012516 21805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:37.012842 21805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:37.013072 21805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:37.020546 21805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:37.020673 21805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:37.020783 21805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:37.020864 21805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:37.020946 21805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:37.030892 21805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44287
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:37.032226 21805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:37.033685 21805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:37.045531 21811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:37.045564 21813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:37.045686 21810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:37.046681 21805 server_base.cc:1034] running on GCE node
I20260924 03:14:37.047597 21805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:37.049227 21805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:37.050450 21805 hybrid_clock.cc:648] HybridClock initialized: now 1790219677050414 us; error 33 us; skew 500 ppm
I20260924 03:14:37.050843 21805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:37 dist-test-slave-5l98 krb5kdc[21792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:37.054404 21805 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:14:37.056105 21805 webserver.cc:460] Webserver started at http://127.0.0.1:36183/ using document root <none> and password file <none>
I20260924 03:14:37.056910 21805 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:37.057056 21805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:37.057379 21805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:37.060474 21805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "f0fc6856fe254e7a8e548cea6dcac530"
format_stamp: "Formatted at 2026-09-24 03:14:37 on dist-test-slave-5l98"
server_key: "4647103c1c8d6733904878bfc05598b2"
server_key_iv: "b148b13f52c67eb8ab74db7a369abc57"
server_key_version: "encryptionkey@0"
I20260924 03:14:37.061378 21805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "f0fc6856fe254e7a8e548cea6dcac530"
format_stamp: "Formatted at 2026-09-24 03:14:37 on dist-test-slave-5l98"
server_key: "4647103c1c8d6733904878bfc05598b2"
server_key_iv: "b148b13f52c67eb8ab74db7a369abc57"
server_key_version: "encryptionkey@0"
I20260924 03:14:37.066879 21805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:14:37.070443 21820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:37.071786 21805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:14:37.072005 21805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "f0fc6856fe254e7a8e548cea6dcac530"
format_stamp: "Formatted at 2026-09-24 03:14:37 on dist-test-slave-5l98"
server_key: "4647103c1c8d6733904878bfc05598b2"
server_key_iv: "b148b13f52c67eb8ab74db7a369abc57"
server_key_version: "encryptionkey@0"
I20260924 03:14:37.072240 21805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:37.092121 21805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:37.097148 21805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:37.097502 21805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:37.117679 21805 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44287
I20260924 03:14:37.117780 21871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44287 every 8 connection(s)
I20260924 03:14:37.119122 21805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:37.123373 21872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:37.129375 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 21805
I20260924 03:14:37.129602 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:37.130890 388 external_mini_cluster.cc:1428] Setting key 6c6d3a1636a74d19ba625295ea7fb298
I20260924 03:14:37.136683 21872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530: Bootstrap starting.
I20260924 03:14:37.140045 21872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:37.141217 21872 log.cc:824] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:37.145184 21872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530: No bootstrap required, opened a new log
Sep 24 03:14:37 dist-test-slave-5l98 krb5kdc[21792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:14:37.154838 21872 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fc6856fe254e7a8e548cea6dcac530" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44287 } }
I20260924 03:14:37.155367 21872 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:37.155511 21872 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0fc6856fe254e7a8e548cea6dcac530, State: Initialized, Role: FOLLOWER
I20260924 03:14:37.156126 21872 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fc6856fe254e7a8e548cea6dcac530" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44287 } }
I20260924 03:14:37.156311 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:37.133158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:49363 (local address 127.0.0.1:44287)
0924 03:14:37.134101 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:37.134118 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:37.134159 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:14:37.135994 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:37.136017 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:37.136456 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:37.136951 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:37.138473 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:37.139838 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:37.141294 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:37.142474 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:37.142692 (+ 218us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:37.148008 (+ 5316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:37.148016 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:37.148035 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:37.148074 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:37.151595 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:37.152551 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:37.152559 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:37.152565 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:37.152739 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:37.153239 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:37.153243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:37.153245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:37.153813 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:37.154162 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:37.154696 (+ 534us) server_negotiation.cc:299] Negotiation successful
0924 03:14:37.155026 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":376,"threads_started":1}
I20260924 03:14:37.156440 21872 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:37.156666 21872 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:37.156879 21872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:37.160053 21872 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fc6856fe254e7a8e548cea6dcac530" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44287 } }
I20260924 03:14:37.160593 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:14:37.160666 21872 leader_election.cc:302] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0fc6856fe254e7a8e548cea6dcac530; no voters:
I20260924 03:14:37.161301 21872 leader_election.cc:288] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:37.161439 21877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:37.163379 21872 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:37.168542 21877 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530 [term 1 LEADER]: Becoming Leader. State: Replica: f0fc6856fe254e7a8e548cea6dcac530, State: Running, Role: LEADER
I20260924 03:14:37.169417 21877 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fc6856fe254e7a8e548cea6dcac530" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44287 } }
I20260924 03:14:37.173570 21878 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0fc6856fe254e7a8e548cea6dcac530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fc6856fe254e7a8e548cea6dcac530" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44287 } } }
I20260924 03:14:37.174077 21878 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:37.174660 21879 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0fc6856fe254e7a8e548cea6dcac530. Latest consensus state: current_term: 1 leader_uuid: "f0fc6856fe254e7a8e548cea6dcac530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fc6856fe254e7a8e548cea6dcac530" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44287 } } }
I20260924 03:14:37.174969 21879 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:37.179098 21892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:37.185250 21892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:37.194551 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:37.172542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:49419 (local address 127.0.0.1:44287)
0924 03:14:37.172746 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:37.172753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:37.172923 (+ 170us) server_negotiation.cc:407] Connection header received
0924 03:14:37.173261 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:37.173275 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:37.173381 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:37.173477 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:37.175838 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:37.177555 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:37.179632 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:37.180433 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:37.180610 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:37.183872 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:37.183877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:37.183881 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:37.183924 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:37.188565 (+ 4641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:37.190101 (+ 1536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:37.190111 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:37.190118 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:37.190318 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:37.191178 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:37.191187 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:37.191193 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:37.191805 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:37.192177 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:37.193697 (+ 1520us) server_negotiation.cc:299] Negotiation successful
0924 03:14:37.193985 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:14:37.206866 21892 catalog_manager.cc:1347] Generated new cluster ID: e89e1c8bfd734ec19d393cac3a4ffcc0
I20260924 03:14:37.206995 21892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:37.241714 21892 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:37.243204 21892 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:37.249332 21892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0fc6856fe254e7a8e548cea6dcac530: Generated new TSK 0
I20260924 03:14:37.250236 21892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:14:37.267504 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 21805
I20260924 03:14:37.268712 21867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:37.369366 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 21805
W20260924 03:14:37.387156 21790 connection.cc:535] client connection to 127.0.0.1:44287 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44287): Connection reset by peer (error 104)
2026-09-24T03:14:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2105 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:37 dist-test-slave-5l98 krb5kdc[21904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:37 dist-test-slave-5l98 krb5kdc[21904](info): set up 2 sockets
Sep 24 03:14:37 dist-test-slave-5l98 krb5kdc[21904](info): commencing operation
krb5kdc: starting...
W20260924 03:14:38.505046 388 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:38 dist-test-slave-5l98 krb5kdc[21904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:14:38.777024 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:39.118867 21917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:39.119240 21917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:39.119418 21917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:39.126747 21917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:39.126947 21917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:39.127279 21917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:39.127413 21917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:39.127496 21917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:39.138512 21917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32955
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:39.140005 21917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:39.141407 21917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:39.154043 21922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:39.153998 21925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:39.154134 21923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:39.154881 21917 server_base.cc:1034] running on GCE node
I20260924 03:14:39.155982 21917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:39.157522 21917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:39.158769 21917 hybrid_clock.cc:648] HybridClock initialized: now 1790219679158696 us; error 70 us; skew 500 ppm
I20260924 03:14:39.159161 21917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:39 dist-test-slave-5l98 krb5kdc[21904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:39.162473 21917 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:14:39.164179 21917 webserver.cc:460] Webserver started at http://127.0.0.1:42131/ using document root <none> and password file <none>
I20260924 03:14:39.164950 21917 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:39.165092 21917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:39.165393 21917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:39.169025 21917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "342f42e2695b4498a53f26292b34e318"
format_stamp: "Formatted at 2026-09-24 03:14:39 on dist-test-slave-5l98"
server_key: "a29d6a79d174aceb8453e1d2d8e808a4"
server_key_iv: "d479350ce6fccc4992894b33c63cbeae"
server_key_version: "encryptionkey@0"
I20260924 03:14:39.169852 21917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "342f42e2695b4498a53f26292b34e318"
format_stamp: "Formatted at 2026-09-24 03:14:39 on dist-test-slave-5l98"
server_key: "a29d6a79d174aceb8453e1d2d8e808a4"
server_key_iv: "d479350ce6fccc4992894b33c63cbeae"
server_key_version: "encryptionkey@0"
I20260924 03:14:39.175462 21917 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:14:39.179227 21932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:39.180764 21917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 03:14:39.180943 21917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "342f42e2695b4498a53f26292b34e318"
format_stamp: "Formatted at 2026-09-24 03:14:39 on dist-test-slave-5l98"
server_key: "a29d6a79d174aceb8453e1d2d8e808a4"
server_key_iv: "d479350ce6fccc4992894b33c63cbeae"
server_key_version: "encryptionkey@0"
I20260924 03:14:39.181165 21917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:39.195895 21917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:39.200595 21917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:39.200899 21917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:39.220250 21917 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32955
I20260924 03:14:39.220268 21983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32955 every 8 connection(s)
I20260924 03:14:39.221736 21917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:39.223496 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 21917
I20260924 03:14:39.223681 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:39.224301 388 external_mini_cluster.cc:1428] Setting key 88b74053fb5e86c1ae79cbf8f2c2228e
I20260924 03:14:39.226598 21984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:39.237823 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:39.226296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:39799 (local address 127.0.0.1:32955)
0924 03:14:39.227151 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:39.227163 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:39.227203 (+ 40us) server_negotiation.cc:407] Connection header received
0924 03:14:39.228971 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:39.228994 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:39.229409 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:39.229835 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:39.230384 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:39.231727 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:39.233209 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:39.233881 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:39.234073 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:39.235418 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:39.235426 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:39.235440 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:39.236038 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:39.236124 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:39.236153 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:39.236339 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:39.236622 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 03:14:39.236811 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":405,"threads_started":1}
I20260924 03:14:39.238469 21984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318: Bootstrap starting.
I20260924 03:14:39.242053 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:14:39.242357 21984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:39.243418 21984 log.cc:824] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:39.246660 21984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318: No bootstrap required, opened a new log
I20260924 03:14:39.254335 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:39.246915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:50055 (local address 127.0.0.1:32955)
0924 03:14:39.247127 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:39.247132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:39.247436 (+ 304us) server_negotiation.cc:407] Connection header received
0924 03:14:39.247636 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:39.247640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:39.247706 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:39.247817 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:39.248535 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:39.249366 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:39.251539 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:39.252045 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:39.252160 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:39.253448 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:39.253453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:39.253457 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:39.253480 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:39.253541 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:39.253557 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:39.253679 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:39.253896 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 03:14:39.253949 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 03:14:39.257508 21984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342f42e2695b4498a53f26292b34e318" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32955 } }
I20260924 03:14:39.257963 21984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:39.258073 21984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342f42e2695b4498a53f26292b34e318, State: Initialized, Role: FOLLOWER
I20260924 03:14:39.258659 21984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342f42e2695b4498a53f26292b34e318" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32955 } }
I20260924 03:14:39.258908 21984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:39.259055 21984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:39.259222 21984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:39.261850 21984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342f42e2695b4498a53f26292b34e318" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32955 } }
I20260924 03:14:39.262377 21984 leader_election.cc:302] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 342f42e2695b4498a53f26292b34e318; no voters:
I20260924 03:14:39.262971 21984 leader_election.cc:288] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:39.263110 21994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:39.263880 21994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318 [term 1 LEADER]: Becoming Leader. State: Replica: 342f42e2695b4498a53f26292b34e318, State: Running, Role: LEADER
I20260924 03:14:39.264477 21994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342f42e2695b4498a53f26292b34e318" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32955 } }
I20260924 03:14:39.265295 21984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:39.267431 21995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "342f42e2695b4498a53f26292b34e318" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342f42e2695b4498a53f26292b34e318" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32955 } } }
I20260924 03:14:39.267405 21996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 342f42e2695b4498a53f26292b34e318. Latest consensus state: current_term: 1 leader_uuid: "342f42e2695b4498a53f26292b34e318" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342f42e2695b4498a53f26292b34e318" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32955 } } }
I20260924 03:14:39.267845 21996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:39.267845 21995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:39.268954 22000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:39.275880 22000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:39.285496 22000 catalog_manager.cc:1347] Generated new cluster ID: de085fcb781145a3b58ad608a76dcca1
I20260924 03:14:39.285648 22000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:39.303853 22000 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:39.305968 22000 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:39.317327 22000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 342f42e2695b4498a53f26292b34e318: Generated new TSK 0
I20260924 03:14:39.318199 22000 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:14:39.329681 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 21917
W20260924 03:14:39.330808 21979 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.89:39799
W20260924 03:14:39.331568 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 21917: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:14:39.331717 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 21917
W20260924 03:14:39.350232 21902 connection.cc:535] client connection to 127.0.0.1:32955 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32955): Connection reset by peer (error 104)
2026-09-24T03:14:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1962 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:39 dist-test-slave-5l98 krb5kdc[22015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:39 dist-test-slave-5l98 krb5kdc[22015](info): set up 2 sockets
Sep 24 03:14:39 dist-test-slave-5l98 krb5kdc[22015](info): commencing operation
krb5kdc: starting...
W20260924 03:14:40.473167 388 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:40 dist-test-slave-5l98 krb5kdc[22015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:14:40.749425 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37487
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:41.088937 22028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:41.089231 22028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:41.089321 22028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:41.096791 22028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:41.096916 22028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:41.096963 22028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:41.097023 22028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:41.097067 22028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:41.107177 22028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37487
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:41.108529 22028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:41.109814 22028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:41.122184 22036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:41.122251 22033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:41.122306 22034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:41.122956 22028 server_base.cc:1034] running on GCE node
I20260924 03:14:41.123798 22028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:41.125344 22028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:41.126595 22028 hybrid_clock.cc:648] HybridClock initialized: now 1790219681126528 us; error 57 us; skew 500 ppm
I20260924 03:14:41.126973 22028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:41 dist-test-slave-5l98 krb5kdc[22015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:41.130151 22028 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:14:41.131729 22028 webserver.cc:460] Webserver started at http://127.0.0.1:35443/ using document root <none> and password file <none>
I20260924 03:14:41.132450 22028 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:41.132637 22028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:41.132946 22028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:41.135979 22028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "e67d4b9d8d7e4eb99178e166b3330e9b"
format_stamp: "Formatted at 2026-09-24 03:14:41 on dist-test-slave-5l98"
server_key: "e512f7ddfa633f2a6e1af5f50bb0af37"
server_key_iv: "75b6f107195cc762d6e7eadb96e3348e"
server_key_version: "encryptionkey@0"
I20260924 03:14:41.136835 22028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "e67d4b9d8d7e4eb99178e166b3330e9b"
format_stamp: "Formatted at 2026-09-24 03:14:41 on dist-test-slave-5l98"
server_key: "e512f7ddfa633f2a6e1af5f50bb0af37"
server_key_iv: "75b6f107195cc762d6e7eadb96e3348e"
server_key_version: "encryptionkey@0"
I20260924 03:14:41.142277 22028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:14:41.145967 22043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:41.147361 22028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:14:41.147560 22028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "e67d4b9d8d7e4eb99178e166b3330e9b"
format_stamp: "Formatted at 2026-09-24 03:14:41 on dist-test-slave-5l98"
server_key: "e512f7ddfa633f2a6e1af5f50bb0af37"
server_key_iv: "75b6f107195cc762d6e7eadb96e3348e"
server_key_version: "encryptionkey@0"
I20260924 03:14:41.147778 22028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:41.153966 22028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:41.158257 22028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:41.158556 22028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:41.178081 22028 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37487
I20260924 03:14:41.178117 22094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37487 every 8 connection(s)
I20260924 03:14:41.179607 22028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:41.183494 22095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:41.185089 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 22028
I20260924 03:14:41.185307 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:41.185853 388 external_mini_cluster.cc:1428] Setting key cf38ddf7d04915004430dfdf219a851d
I20260924 03:14:41.195099 22098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:41.188142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:43307 (local address 127.0.0.1:37487)
0924 03:14:41.189525 (+ 1383us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:41.189537 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:41.189596 (+ 59us) server_negotiation.cc:407] Connection header received
0924 03:14:41.191285 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:41.191308 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:41.191664 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:41.192159 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:41.193310 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:41.193322 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:41.193338 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:41.193356 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:41.193425 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:41.193446 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:41.193722 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:41.193872 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 03:14:41.194065 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1147,"thread_start_us":855,"threads_started":1}
I20260924 03:14:41.196508 22095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b: Bootstrap starting.
I20260924 03:14:41.199193 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:14:41.200171 22095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:41.201278 22095 log.cc:824] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:41.204296 22095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b: No bootstrap required, opened a new log
I20260924 03:14:41.209555 22098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:41.206761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:52863 (local address 127.0.0.1:37487)
0924 03:14:41.207005 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:41.207012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:41.207354 (+ 342us) server_negotiation.cc:407] Connection header received
0924 03:14:41.207535 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:41.207540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:41.207597 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:41.207700 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:41.208802 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:41.208809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:41.208818 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:41.208841 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:41.208887 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:41.208906 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:41.209018 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:41.209166 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 03:14:41.209223 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 03:14:41.214648 22095 raft_consensus.cc:348] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e67d4b9d8d7e4eb99178e166b3330e9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37487 } }
I20260924 03:14:41.215097 22095 raft_consensus.cc:374] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:41.215179 22095 raft_consensus.cc:729] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e67d4b9d8d7e4eb99178e166b3330e9b, State: Initialized, Role: FOLLOWER
I20260924 03:14:41.215843 22095 consensus_queue.cc:260] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e67d4b9d8d7e4eb99178e166b3330e9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37487 } }
I20260924 03:14:41.216081 22095 raft_consensus.cc:388] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:41.216189 22095 raft_consensus.cc:482] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:41.216331 22095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:41.219043 22095 raft_consensus.cc:504] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e67d4b9d8d7e4eb99178e166b3330e9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37487 } }
I20260924 03:14:41.219554 22095 leader_election.cc:302] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e67d4b9d8d7e4eb99178e166b3330e9b; no voters:
I20260924 03:14:41.220189 22095 leader_election.cc:288] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:41.220379 22105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:41.221072 22105 raft_consensus.cc:686] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b [term 1 LEADER]: Becoming Leader. State: Replica: e67d4b9d8d7e4eb99178e166b3330e9b, State: Running, Role: LEADER
I20260924 03:14:41.221632 22105 consensus_queue.cc:237] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e67d4b9d8d7e4eb99178e166b3330e9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37487 } }
I20260924 03:14:41.222476 22095 sys_catalog.cc:564] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:41.224709 22107 sys_catalog.cc:455] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e67d4b9d8d7e4eb99178e166b3330e9b. Latest consensus state: current_term: 1 leader_uuid: "e67d4b9d8d7e4eb99178e166b3330e9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e67d4b9d8d7e4eb99178e166b3330e9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37487 } } }
I20260924 03:14:41.225243 22107 sys_catalog.cc:458] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:41.227903 22106 sys_catalog.cc:455] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e67d4b9d8d7e4eb99178e166b3330e9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e67d4b9d8d7e4eb99178e166b3330e9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37487 } } }
I20260924 03:14:41.228380 22106 sys_catalog.cc:458] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:41.232574 22113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:41.237149 22113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:41.246639 22113 catalog_manager.cc:1347] Generated new cluster ID: a56ade3eb30644f497112e6acd47ec77
I20260924 03:14:41.246795 22113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:41.264178 22113 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:41.264744 22113 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:41.279940 22113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e67d4b9d8d7e4eb99178e166b3330e9b: Generated new TSK 0
I20260924 03:14:41.281066 22113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:14:41.290321 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 22028
W20260924 03:14:41.291494 22090 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.89:43307
W20260924 03:14:41.292351 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 22028: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:14:41.292569 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 22028
2026-09-24T03:14:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1959 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:41 dist-test-slave-5l98 krb5kdc[22126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:41 dist-test-slave-5l98 krb5kdc[22126](info): set up 2 sockets
Sep 24 03:14:41 dist-test-slave-5l98 krb5kdc[22126](info): commencing operation
krb5kdc: starting...
W20260924 03:14:42.409132 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:42 dist-test-slave-5l98 krb5kdc[22126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:14:42.683785 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39285
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:43.026556 22139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:43.026896 22139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:43.026994 22139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:43.034422 22139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:43.034546 22139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:43.034593 22139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:43.034651 22139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:43.034695 22139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:43.044950 22139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39285
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:43.046227 22139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:43.047494 22139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:43.060220 22147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:43.060264 22145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:43.060509 22144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:43.061573 22139 server_base.cc:1034] running on GCE node
I20260924 03:14:43.062393 22139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:43.063855 22139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:43.065073 22139 hybrid_clock.cc:648] HybridClock initialized: now 1790219683065001 us; error 69 us; skew 500 ppm
I20260924 03:14:43.065419 22139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:43 dist-test-slave-5l98 krb5kdc[22126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:43.068742 22139 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:14:43.070442 22139 webserver.cc:460] Webserver started at http://127.0.97.62:38503/ using document root <none> and password file <none>
I20260924 03:14:43.071174 22139 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:43.071322 22139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:43.071626 22139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:43.074748 22139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "7637779832cc46af8d3169bacb535f21"
format_stamp: "Formatted at 2026-09-24 03:14:43 on dist-test-slave-5l98"
server_key: "e90a3a26b8f9361c251d324e36e1e743"
server_key_iv: "6d2eafe71e927ce9b447b6187ea2a05d"
server_key_version: "encryptionkey@0"
I20260924 03:14:43.075523 22139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "7637779832cc46af8d3169bacb535f21"
format_stamp: "Formatted at 2026-09-24 03:14:43 on dist-test-slave-5l98"
server_key: "e90a3a26b8f9361c251d324e36e1e743"
server_key_iv: "6d2eafe71e927ce9b447b6187ea2a05d"
server_key_version: "encryptionkey@0"
I20260924 03:14:43.081153 22139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:14:43.084806 22154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:43.086210 22139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:14:43.086404 22139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "7637779832cc46af8d3169bacb535f21"
format_stamp: "Formatted at 2026-09-24 03:14:43 on dist-test-slave-5l98"
server_key: "e90a3a26b8f9361c251d324e36e1e743"
server_key_iv: "6d2eafe71e927ce9b447b6187ea2a05d"
server_key_version: "encryptionkey@0"
I20260924 03:14:43.086622 22139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:43.106806 22139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:43.111219 22139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:43.111526 22139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:43.131520 22139 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39285
I20260924 03:14:43.131543 22205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39285 every 8 connection(s)
I20260924 03:14:43.132961 22139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:43.137054 22206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:43.140758 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 22139
I20260924 03:14:43.140940 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:43.141392 388 external_mini_cluster.cc:1428] Setting key c320100c92d31c360f3718641ccbcd69
I20260924 03:14:43.150691 22206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21: Bootstrap starting.
I20260924 03:14:43.154484 22206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:43.155542 22206 log.cc:824] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:14:43 dist-test-slave-5l98 krb5kdc[22126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:14:43.159037 22206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21: No bootstrap required, opened a new log
I20260924 03:14:43.168236 22209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:43.143319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38353 (local address 127.0.97.62:39285)
0924 03:14:43.144118 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:43.144135 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:43.144186 (+ 51us) server_negotiation.cc:407] Connection header received
0924 03:14:43.146313 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:43.146355 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:43.146911 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:43.147369 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:43.148750 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:43.150718 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:43.152109 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:43.153214 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:43.153414 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:43.158845 (+ 5431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:43.158857 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:43.158878 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:43.158925 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:43.163401 (+ 4476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:43.164274 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:43.164284 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:43.164299 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:43.164613 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:43.165209 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:43.165212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:43.165215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:43.165790 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:43.166067 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:43.166732 (+ 665us) server_negotiation.cc:299] Negotiation successful
0924 03:14:43.167133 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":320,"threads_started":1}
I20260924 03:14:43.170166 22206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7637779832cc46af8d3169bacb535f21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39285 } }
I20260924 03:14:43.170584 22206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:43.170722 22206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7637779832cc46af8d3169bacb535f21, State: Initialized, Role: FOLLOWER
I20260924 03:14:43.171449 22206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7637779832cc46af8d3169bacb535f21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39285 } }
I20260924 03:14:43.171756 22206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:43.171926 22206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:43.172120 22206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:43.172711 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:14:43.175745 22206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7637779832cc46af8d3169bacb535f21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39285 } }
I20260924 03:14:43.176584 22206 leader_election.cc:302] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7637779832cc46af8d3169bacb535f21; no voters:
I20260924 03:14:43.177424 22206 leader_election.cc:288] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:43.177558 22214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:43.179203 22214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21 [term 1 LEADER]: Becoming Leader. State: Replica: 7637779832cc46af8d3169bacb535f21, State: Running, Role: LEADER
I20260924 03:14:43.180064 22206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:43.179948 22214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7637779832cc46af8d3169bacb535f21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39285 } }
I20260924 03:14:43.184927 22216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7637779832cc46af8d3169bacb535f21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7637779832cc46af8d3169bacb535f21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39285 } } }
I20260924 03:14:43.185464 22216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:43.185304 22217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7637779832cc46af8d3169bacb535f21. Latest consensus state: current_term: 1 leader_uuid: "7637779832cc46af8d3169bacb535f21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7637779832cc46af8d3169bacb535f21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39285 } } }
I20260924 03:14:43.185928 22217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:43.194075 22228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:43.200086 22228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:43.203276 22209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:43.179284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58267 (local address 127.0.97.62:39285)
0924 03:14:43.179508 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:43.179516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:43.180061 (+ 545us) server_negotiation.cc:407] Connection header received
0924 03:14:43.180273 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:43.180279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:43.180364 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:43.180446 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:43.183127 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:43.184758 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:43.187020 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:43.189483 (+ 2463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:43.189668 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:43.194159 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:43.194168 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:43.194177 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:43.194227 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:43.198534 (+ 4307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:43.199564 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:43.199570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:43.199574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:43.199811 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:43.200344 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:43.200350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:43.200358 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:43.200843 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:43.201136 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:43.202643 (+ 1507us) server_negotiation.cc:299] Negotiation successful
0924 03:14:43.202847 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 03:14:43.210582 22228 catalog_manager.cc:1347] Generated new cluster ID: 3950fbb0d9094552a3fe31499548b129
I20260924 03:14:43.210718 22228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:43.238843 22228 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:43.240242 22228 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:43.258834 22228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7637779832cc46af8d3169bacb535f21: Generated new TSK 0
I20260924 03:14:43.259923 22228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:14:43.279066 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 22139
I20260924 03:14:43.280432 22201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:43.378407 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 22139
W20260924 03:14:43.396538 22124 connection.cc:535] client connection to 127.0.97.62:39285 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:39285): Connection reset by peer (error 104)
2026-09-24T03:14:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2087 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:43 dist-test-slave-5l98 krb5kdc[22238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:43 dist-test-slave-5l98 krb5kdc[22238](info): set up 2 sockets
Sep 24 03:14:43 dist-test-slave-5l98 krb5kdc[22238](info): commencing operation
krb5kdc: starting...
W20260924 03:14:44.529162 388 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:44 dist-test-slave-5l98 krb5kdc[22238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:14:44.800815 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35135
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:45.141855 22251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:45.142154 22251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:45.142247 22251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:45.149811 22251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:45.149931 22251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:45.149977 22251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:45.150030 22251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:45.150079 22251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:45.160413 22251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35135
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:45.161763 22251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:45.163024 22251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:45.176343 22259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:45.176357 22257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:45.176533 22256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:45.177187 22251 server_base.cc:1034] running on GCE node
I20260924 03:14:45.178117 22251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:45.179672 22251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:45.180919 22251 hybrid_clock.cc:648] HybridClock initialized: now 1790219685180860 us; error 53 us; skew 500 ppm
I20260924 03:14:45.181293 22251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:45 dist-test-slave-5l98 krb5kdc[22238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:45.184613 22251 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:14:45.186323 22251 webserver.cc:460] Webserver started at http://127.0.97.62:36011/ using document root <none> and password file <none>
I20260924 03:14:45.187021 22251 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:45.187165 22251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:45.187479 22251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:45.190598 22251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "98e05a0e809d4b43977de9117e5c0123"
format_stamp: "Formatted at 2026-09-24 03:14:45 on dist-test-slave-5l98"
server_key: "f86166d89997ef96d6c9d51c621a9659"
server_key_iv: "acbb9c541ab4219d2b18337dc1cfb825"
server_key_version: "encryptionkey@0"
I20260924 03:14:45.191390 22251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "98e05a0e809d4b43977de9117e5c0123"
format_stamp: "Formatted at 2026-09-24 03:14:45 on dist-test-slave-5l98"
server_key: "f86166d89997ef96d6c9d51c621a9659"
server_key_iv: "acbb9c541ab4219d2b18337dc1cfb825"
server_key_version: "encryptionkey@0"
I20260924 03:14:45.196905 22251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 03:14:45.200426 22266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:45.201928 22251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:14:45.202116 22251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "98e05a0e809d4b43977de9117e5c0123"
format_stamp: "Formatted at 2026-09-24 03:14:45 on dist-test-slave-5l98"
server_key: "f86166d89997ef96d6c9d51c621a9659"
server_key_iv: "acbb9c541ab4219d2b18337dc1cfb825"
server_key_version: "encryptionkey@0"
I20260924 03:14:45.202353 22251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:45.232393 22251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:45.237049 22251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:45.237363 22251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:45.257057 22251 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35135
I20260924 03:14:45.257112 22317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35135 every 8 connection(s)
I20260924 03:14:45.258492 22251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:45.262380 22318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:45.267170 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 22251
I20260924 03:14:45.267392 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:45.267822 388 external_mini_cluster.cc:1428] Setting key d24b4cf2b3bdc5bcfce3ff364830bc73
I20260924 03:14:45.274052 22318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123: Bootstrap starting.
I20260924 03:14:45.278345 22318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:45.279763 22318 log.cc:824] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:45.282606 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:45.269637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60103 (local address 127.0.97.62:35135)
0924 03:14:45.270450 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:45.270487 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:45.270529 (+ 42us) server_negotiation.cc:407] Connection header received
0924 03:14:45.272339 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:45.272364 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:45.272845 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:45.273179 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:45.273852 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:45.275241 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:45.277469 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:45.278165 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:45.278334 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:45.279645 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:45.279652 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:45.279667 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:45.280358 (+ 691us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:45.280469 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:45.280536 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:45.280730 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:45.281158 (+ 428us) server_negotiation.cc:299] Negotiation successful
0924 03:14:45.281355 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":387,"threads_started":1}
I20260924 03:14:45.283279 22318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123: No bootstrap required, opened a new log
I20260924 03:14:45.286953 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:14:45.291797 22318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e05a0e809d4b43977de9117e5c0123" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35135 } }
I20260924 03:14:45.292321 22318 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:45.292441 22318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98e05a0e809d4b43977de9117e5c0123, State: Initialized, Role: FOLLOWER
I20260924 03:14:45.293183 22318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e05a0e809d4b43977de9117e5c0123" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35135 } }
I20260924 03:14:45.293504 22318 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:45.293661 22318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:45.293853 22318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:45.296566 22318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e05a0e809d4b43977de9117e5c0123" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35135 } }
I20260924 03:14:45.297168 22318 leader_election.cc:302] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98e05a0e809d4b43977de9117e5c0123; no voters:
I20260924 03:14:45.297930 22318 leader_election.cc:288] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:45.298079 22328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:45.299044 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:45.291464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57975 (local address 127.0.97.62:35135)
0924 03:14:45.291646 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:45.291655 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:45.292092 (+ 437us) server_negotiation.cc:407] Connection header received
0924 03:14:45.292357 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:45.292364 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:45.292456 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:45.292610 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:45.293141 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:45.294308 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:45.295720 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:45.296410 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:45.296599 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:45.297916 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:45.297924 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:45.297930 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:45.297958 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:45.298004 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:45.298025 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:45.298145 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:45.298440 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 03:14:45.298518 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 03:14:45.299276 22328 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123 [term 1 LEADER]: Becoming Leader. State: Replica: 98e05a0e809d4b43977de9117e5c0123, State: Running, Role: LEADER
I20260924 03:14:45.299947 22328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e05a0e809d4b43977de9117e5c0123" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35135 } }
I20260924 03:14:45.302381 22318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:45.302872 22329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98e05a0e809d4b43977de9117e5c0123. Latest consensus state: current_term: 1 leader_uuid: "98e05a0e809d4b43977de9117e5c0123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e05a0e809d4b43977de9117e5c0123" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35135 } } }
I20260924 03:14:45.303270 22329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:45.303814 22330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98e05a0e809d4b43977de9117e5c0123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e05a0e809d4b43977de9117e5c0123" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35135 } } }
I20260924 03:14:45.304220 22330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:45.309710 22332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:45.317377 22332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:45.327622 22332 catalog_manager.cc:1347] Generated new cluster ID: 363923097fe94f888edca707d2249816
I20260924 03:14:45.327764 22332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:45.348549 22332 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:45.349907 22332 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:45.364391 22332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98e05a0e809d4b43977de9117e5c0123: Generated new TSK 0
I20260924 03:14:45.365355 22332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:14:45.375685 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 22251
W20260924 03:14:45.377120 22313 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60103
W20260924 03:14:45.377978 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 22251: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:14:45.378145 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 22251
W20260924 03:14:45.393241 22236 connection.cc:535] client connection to 127.0.97.62:35135 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:35135): Connection reset by peer (error 104)
2026-09-24T03:14:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1996 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:45 dist-test-slave-5l98 krb5kdc[22349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:45 dist-test-slave-5l98 krb5kdc[22349](info): set up 2 sockets
Sep 24 03:14:45 dist-test-slave-5l98 krb5kdc[22349](info): commencing operation
krb5kdc: starting...
W20260924 03:14:46.508981 388 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.010s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:46 dist-test-slave-5l98 krb5kdc[22349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:14:46.784991 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34633
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:47.124851 22362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:47.125145 22362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:47.125329 22362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:47.132652 22362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:47.132773 22362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:47.132889 22362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:47.132969 22362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:47.133042 22362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:47.143123 22362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34633
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:47.144438 22362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:47.145825 22362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:47.161883 22367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:47.161989 22370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:47.162019 22368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:47.163192 22362 server_base.cc:1034] running on GCE node
I20260924 03:14:47.164099 22362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:47.165664 22362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:47.166925 22362 hybrid_clock.cc:648] HybridClock initialized: now 1790219687166868 us; error 58 us; skew 500 ppm
I20260924 03:14:47.167289 22362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:47 dist-test-slave-5l98 krb5kdc[22349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:47.170612 22362 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:14:47.172276 22362 webserver.cc:460] Webserver started at http://127.0.97.62:41099/ using document root <none> and password file <none>
I20260924 03:14:47.173017 22362 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:47.173121 22362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:47.173372 22362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:47.176394 22362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "a51ead06e89b4124b70390f5b6498c0b"
format_stamp: "Formatted at 2026-09-24 03:14:47 on dist-test-slave-5l98"
server_key: "34aa93e4197ac4457523bb0dbfcb4018"
server_key_iv: "1bace40cf9a1ebd17941ea33669c701a"
server_key_version: "encryptionkey@0"
I20260924 03:14:47.177248 22362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "a51ead06e89b4124b70390f5b6498c0b"
format_stamp: "Formatted at 2026-09-24 03:14:47 on dist-test-slave-5l98"
server_key: "34aa93e4197ac4457523bb0dbfcb4018"
server_key_iv: "1bace40cf9a1ebd17941ea33669c701a"
server_key_version: "encryptionkey@0"
I20260924 03:14:47.182634 22362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 03:14:47.186335 22377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:47.187737 22362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:14:47.187927 22362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "a51ead06e89b4124b70390f5b6498c0b"
format_stamp: "Formatted at 2026-09-24 03:14:47 on dist-test-slave-5l98"
server_key: "34aa93e4197ac4457523bb0dbfcb4018"
server_key_iv: "1bace40cf9a1ebd17941ea33669c701a"
server_key_version: "encryptionkey@0"
I20260924 03:14:47.188104 22362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:47.194572 22362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:47.199414 22362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:47.199702 22362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:47.220048 22362 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34633
I20260924 03:14:47.220093 22428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34633 every 8 connection(s)
I20260924 03:14:47.221809 22362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:47.226044 22429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:47.232160 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 22362
I20260924 03:14:47.232393 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:47.232947 388 external_mini_cluster.cc:1428] Setting key 1e80b9ce3350ee6f5f09912795e16a32
I20260924 03:14:47.238907 22429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b: Bootstrap starting.
I20260924 03:14:47.242765 22432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:47.235076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43177 (local address 127.0.97.62:34633)
0924 03:14:47.236073 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:47.236097 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:47.236155 (+ 58us) server_negotiation.cc:407] Connection header received
0924 03:14:47.238589 (+ 2434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:47.238621 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:47.239097 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:47.239557 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:47.240780 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:47.240789 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:47.240814 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:47.240837 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:47.240925 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:47.240953 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:47.241201 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:47.241327 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 03:14:47.241531 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":419,"threads_started":1}
I20260924 03:14:47.243090 22429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:47.244433 22429 log.cc:824] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:47.247155 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:14:47.247644 22429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b: No bootstrap required, opened a new log
I20260924 03:14:47.255275 22432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:47.252528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44995 (local address 127.0.97.62:34633)
0924 03:14:47.252728 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:47.252733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:47.253147 (+ 414us) server_negotiation.cc:407] Connection header received
0924 03:14:47.253354 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:47.253359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:47.253419 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:47.253528 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:47.254462 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:47.254466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:47.254470 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:47.254484 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:47.254527 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:47.254542 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:47.254652 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:47.254887 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 03:14:47.254987 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 03:14:47.256345 22429 raft_consensus.cc:348] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51ead06e89b4124b70390f5b6498c0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34633 } }
I20260924 03:14:47.256788 22429 raft_consensus.cc:374] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:47.256899 22429 raft_consensus.cc:729] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51ead06e89b4124b70390f5b6498c0b, State: Initialized, Role: FOLLOWER
I20260924 03:14:47.257475 22429 consensus_queue.cc:260] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51ead06e89b4124b70390f5b6498c0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34633 } }
I20260924 03:14:47.257717 22429 raft_consensus.cc:388] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:47.257859 22429 raft_consensus.cc:482] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:47.258031 22429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:47.260569 22429 raft_consensus.cc:504] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51ead06e89b4124b70390f5b6498c0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34633 } }
I20260924 03:14:47.261123 22429 leader_election.cc:302] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a51ead06e89b4124b70390f5b6498c0b; no voters:
I20260924 03:14:47.261703 22429 leader_election.cc:288] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:47.261869 22439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:47.262598 22439 raft_consensus.cc:686] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b [term 1 LEADER]: Becoming Leader. State: Replica: a51ead06e89b4124b70390f5b6498c0b, State: Running, Role: LEADER
I20260924 03:14:47.263230 22439 consensus_queue.cc:237] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51ead06e89b4124b70390f5b6498c0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34633 } }
I20260924 03:14:47.263844 22429 sys_catalog.cc:564] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:47.266595 22440 sys_catalog.cc:455] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a51ead06e89b4124b70390f5b6498c0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51ead06e89b4124b70390f5b6498c0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34633 } } }
I20260924 03:14:47.267315 22440 sys_catalog.cc:458] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:47.266736 22441 sys_catalog.cc:455] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a51ead06e89b4124b70390f5b6498c0b. Latest consensus state: current_term: 1 leader_uuid: "a51ead06e89b4124b70390f5b6498c0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51ead06e89b4124b70390f5b6498c0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34633 } } }
I20260924 03:14:47.268333 22441 sys_catalog.cc:458] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:47.272959 22448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:47.277359 22448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:47.286602 22448 catalog_manager.cc:1347] Generated new cluster ID: 7a61f011d25241c3a6ccaa107bc26df6
I20260924 03:14:47.286729 22448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:47.305594 22448 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:47.306053 22448 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:47.318918 22448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a51ead06e89b4124b70390f5b6498c0b: Generated new TSK 0
I20260924 03:14:47.320070 22448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:14:47.333101 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 22362
W20260924 03:14:47.334393 22424 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43177
W20260924 03:14:47.335222 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 22362: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:14:47.335391 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 22362
2026-09-24T03:14:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1960 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:47 dist-test-slave-5l98 krb5kdc[22460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:47 dist-test-slave-5l98 krb5kdc[22460](info): set up 2 sockets
Sep 24 03:14:47 dist-test-slave-5l98 krb5kdc[22460](info): commencing operation
krb5kdc: starting...
W20260924 03:14:48.481117 388 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.006s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:48 dist-test-slave-5l98 krb5kdc[22460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:14:48.753808 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43783
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:49.097425 22473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:49.097747 22473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:49.097849 22473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:49.105381 22473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:49.105500 22473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:49.105546 22473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:49.105614 22473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:49.105659 22473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:49.115713 22473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:49.117041 22473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:49.118357 22473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:49.130796 22478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:49.130796 22481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:49.130795 22479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:49.131438 22473 server_base.cc:1034] running on GCE node
I20260924 03:14:49.132236 22473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:49.133744 22473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:49.134984 22473 hybrid_clock.cc:648] HybridClock initialized: now 1790219689134923 us; error 47 us; skew 500 ppm
I20260924 03:14:49.135346 22473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:49 dist-test-slave-5l98 krb5kdc[22460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:49.138828 22473 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:14:49.140447 22473 webserver.cc:460] Webserver started at http://127.0.97.62:43819/ using document root <none> and password file <none>
I20260924 03:14:49.141251 22473 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:49.141377 22473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:49.141716 22473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:49.144796 22473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "037c3a27a7f047a8813e4b44eed7d71c"
format_stamp: "Formatted at 2026-09-24 03:14:49 on dist-test-slave-5l98"
server_key: "a3fb3e48a181e03239bdc0135ac9e5e4"
server_key_iv: "8efd88d9d8b090558c794cf4cf05f001"
server_key_version: "encryptionkey@0"
I20260924 03:14:49.145565 22473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "037c3a27a7f047a8813e4b44eed7d71c"
format_stamp: "Formatted at 2026-09-24 03:14:49 on dist-test-slave-5l98"
server_key: "a3fb3e48a181e03239bdc0135ac9e5e4"
server_key_iv: "8efd88d9d8b090558c794cf4cf05f001"
server_key_version: "encryptionkey@0"
I20260924 03:14:49.150938 22473 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 03:14:49.154845 22488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:49.156244 22473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:14:49.156452 22473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "037c3a27a7f047a8813e4b44eed7d71c"
format_stamp: "Formatted at 2026-09-24 03:14:49 on dist-test-slave-5l98"
server_key: "a3fb3e48a181e03239bdc0135ac9e5e4"
server_key_iv: "8efd88d9d8b090558c794cf4cf05f001"
server_key_version: "encryptionkey@0"
I20260924 03:14:49.156854 22473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:49.185092 22473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:49.189664 22473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:49.189985 22473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:49.210032 22473 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43783
I20260924 03:14:49.210089 22539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43783 every 8 connection(s)
I20260924 03:14:49.211493 22473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:49.215348 22540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:49.220995 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 22473
I20260924 03:14:49.221189 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:49.221722 388 external_mini_cluster.cc:1428] Setting key 89d114628babca181397ea3970e3cfce
I20260924 03:14:49.226895 22540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c: Bootstrap starting.
I20260924 03:14:49.230813 22540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:49.232172 22540 log.cc:824] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:49.236313 22540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c: No bootstrap required, opened a new log
Sep 24 03:14:49 dist-test-slave-5l98 krb5kdc[22460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:14:49.244745 22540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "037c3a27a7f047a8813e4b44eed7d71c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } }
I20260924 03:14:49.245173 22540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:49.245302 22540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 037c3a27a7f047a8813e4b44eed7d71c, State: Initialized, Role: FOLLOWER
I20260924 03:14:49.245978 22540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "037c3a27a7f047a8813e4b44eed7d71c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } }
I20260924 03:14:49.246249 22540 raft_consensus.cc:388] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:49.246420 22540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:49.246628 22540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:49.248238 22543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:49.223729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47291 (local address 127.0.97.62:43783)
0924 03:14:49.224708 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:49.224721 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:49.224776 (+ 55us) server_negotiation.cc:407] Connection header received
0924 03:14:49.226610 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:49.226634 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:49.227114 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:49.227461 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:49.229433 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:49.231005 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:49.232864 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:49.233974 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:49.234126 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:49.239334 (+ 5208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:49.239348 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:49.239367 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:49.239415 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:49.243253 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:49.244215 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:49.244223 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:49.244230 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:49.244423 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:49.244914 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:49.244918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:49.244920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:49.245663 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:49.246032 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:49.246501 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 03:14:49.247039 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":485,"threads_started":1}
I20260924 03:14:49.249473 22540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "037c3a27a7f047a8813e4b44eed7d71c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } }
I20260924 03:14:49.250041 22540 leader_election.cc:302] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 037c3a27a7f047a8813e4b44eed7d71c; no voters:
I20260924 03:14:49.250701 22540 leader_election.cc:288] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:49.250852 22545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:49.252907 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:14:49.252853 22545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c [term 1 LEADER]: Becoming Leader. State: Replica: 037c3a27a7f047a8813e4b44eed7d71c, State: Running, Role: LEADER
I20260924 03:14:49.253460 22545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "037c3a27a7f047a8813e4b44eed7d71c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } }
I20260924 03:14:49.253758 22540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:49.256139 22547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 037c3a27a7f047a8813e4b44eed7d71c. Latest consensus state: current_term: 1 leader_uuid: "037c3a27a7f047a8813e4b44eed7d71c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "037c3a27a7f047a8813e4b44eed7d71c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } } }
I20260924 03:14:49.256577 22547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:49.257140 22546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "037c3a27a7f047a8813e4b44eed7d71c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "037c3a27a7f047a8813e4b44eed7d71c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43783 } } }
I20260924 03:14:49.257514 22546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:49.259960 22551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:49.267598 22551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:49.289681 22551 catalog_manager.cc:1347] Generated new cluster ID: 5ad98eb25b134d319ee9e921b80a1cd7
I20260924 03:14:49.289891 22551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:49.295440 22543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:49.264750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43843 (local address 127.0.97.62:43783)
0924 03:14:49.264936 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:49.264943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:49.275492 (+ 10549us) server_negotiation.cc:407] Connection header received
0924 03:14:49.276010 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:49.276017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:49.276101 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:49.276181 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:49.278710 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:49.280235 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:49.282818 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:49.283817 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:49.283976 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:49.288051 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:49.288058 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:49.288063 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:49.288106 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:49.292273 (+ 4167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:49.293154 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:49.293160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:49.293164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:49.293320 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:49.293742 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:49.293747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:49.293750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:49.294113 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:49.294321 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:49.294858 (+ 537us) server_negotiation.cc:299] Negotiation successful
0924 03:14:49.295074 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:14:49.301517 22551 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:49.303278 22551 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:49.320859 22551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 037c3a27a7f047a8813e4b44eed7d71c: Generated new TSK 0
I20260924 03:14:49.321776 22551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:14:49.328359 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 22473
I20260924 03:14:49.329576 22535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:49.428278 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 22473
2026-09-24T03:14:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2091 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:49 dist-test-slave-5l98 krb5kdc[22573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:49 dist-test-slave-5l98 krb5kdc[22573](info): set up 2 sockets
Sep 24 03:14:49 dist-test-slave-5l98 krb5kdc[22573](info): commencing operation
krb5kdc: starting...
W20260924 03:14:50.557085 388 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:50 dist-test-slave-5l98 krb5kdc[22573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:14:50.836946 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42933
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:51.184751 22586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:51.185170 22586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:51.185369 22586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:51.194267 22586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:51.194478 22586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:51.194598 22586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:51.194722 22586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:51.194810 22586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:51.205700 22586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:51.207216 22586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:51.208700 22586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:51.221957 22591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:51.222036 22592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:51.222036 22594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:51.223762 22586 server_base.cc:1034] running on GCE node
I20260924 03:14:51.224858 22586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:51.226558 22586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:51.227860 22586 hybrid_clock.cc:648] HybridClock initialized: now 1790219691227798 us; error 96 us; skew 500 ppm
I20260924 03:14:51.228284 22586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:51 dist-test-slave-5l98 krb5kdc[22573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:51.231810 22586 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:14:51.234161 22586 webserver.cc:460] Webserver started at http://127.0.97.62:38063/ using document root <none> and password file <none>
I20260924 03:14:51.235176 22586 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:51.235342 22586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:51.235718 22586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:51.239260 22586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "6015dc94dfe2414492bd7903325b5729"
format_stamp: "Formatted at 2026-09-24 03:14:51 on dist-test-slave-5l98"
server_key: "ae9d07c38aae5d76c9af2ca01f6bc105"
server_key_iv: "c7b7805932fc584f97d358e032f1a8fa"
server_key_version: "encryptionkey@0"
I20260924 03:14:51.240043 22586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "6015dc94dfe2414492bd7903325b5729"
format_stamp: "Formatted at 2026-09-24 03:14:51 on dist-test-slave-5l98"
server_key: "ae9d07c38aae5d76c9af2ca01f6bc105"
server_key_iv: "c7b7805932fc584f97d358e032f1a8fa"
server_key_version: "encryptionkey@0"
I20260924 03:14:51.245872 22586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 03:14:51.249993 22601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:51.251571 22586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:14:51.251873 22586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "6015dc94dfe2414492bd7903325b5729"
format_stamp: "Formatted at 2026-09-24 03:14:51 on dist-test-slave-5l98"
server_key: "ae9d07c38aae5d76c9af2ca01f6bc105"
server_key_iv: "c7b7805932fc584f97d358e032f1a8fa"
server_key_version: "encryptionkey@0"
I20260924 03:14:51.252135 22586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:51.280344 22586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:51.285671 22586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:51.286027 22586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:51.307524 22586 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42933
I20260924 03:14:51.307547 22652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42933 every 8 connection(s)
I20260924 03:14:51.309142 22586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:51.313218 22653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:51.314352 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 22586
I20260924 03:14:51.314563 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:51.315037 388 external_mini_cluster.cc:1428] Setting key 84b72de9a084775ce385068a3541eb2f
I20260924 03:14:51.328780 22653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729: Bootstrap starting.
I20260924 03:14:51.331192 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:51.317097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49077 (local address 127.0.97.62:42933)
0924 03:14:51.317919 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:51.317934 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:51.317982 (+ 48us) server_negotiation.cc:407] Connection header received
0924 03:14:51.320470 (+ 2488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:51.320529 (+ 59us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:51.321121 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:51.321615 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:51.322223 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:51.324091 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:51.325666 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:51.326782 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:51.326999 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:51.328284 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:51.328297 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:51.328320 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:51.329077 (+ 757us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:51.329181 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:51.329213 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:51.329341 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:51.329593 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 03:14:51.329860 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":350,"threads_started":1}
I20260924 03:14:51.332806 22653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:51.334235 22653 log.cc:824] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:51.335939 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:14:51.337678 22653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729: No bootstrap required, opened a new log
I20260924 03:14:51.347921 22653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6015dc94dfe2414492bd7903325b5729" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42933 } }
I20260924 03:14:51.348528 22653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:51.348683 22653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6015dc94dfe2414492bd7903325b5729, State: Initialized, Role: FOLLOWER
I20260924 03:14:51.348951 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:51.341629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46525 (local address 127.0.97.62:42933)
0924 03:14:51.341824 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:51.341831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:51.342164 (+ 333us) server_negotiation.cc:407] Connection header received
0924 03:14:51.342329 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:51.342334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:51.342413 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:51.342519 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:51.343136 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:51.344311 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:51.345845 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:51.346575 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:51.346711 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:51.347903 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:51.347910 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:51.347916 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:51.347937 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:51.347985 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:51.348006 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:51.348106 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:51.348329 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 03:14:51.348430 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 03:14:51.349416 22653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6015dc94dfe2414492bd7903325b5729" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42933 } }
I20260924 03:14:51.349754 22653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:51.349921 22653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:51.350111 22653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:51.352941 22653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6015dc94dfe2414492bd7903325b5729" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42933 } }
I20260924 03:14:51.353506 22653 leader_election.cc:302] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6015dc94dfe2414492bd7903325b5729; no voters:
I20260924 03:14:51.354154 22653 leader_election.cc:288] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:51.354372 22663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:51.355156 22663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729 [term 1 LEADER]: Becoming Leader. State: Replica: 6015dc94dfe2414492bd7903325b5729, State: Running, Role: LEADER
I20260924 03:14:51.355839 22663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6015dc94dfe2414492bd7903325b5729" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42933 } }
I20260924 03:14:51.356292 22653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:51.358704 22664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6015dc94dfe2414492bd7903325b5729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6015dc94dfe2414492bd7903325b5729" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42933 } } }
I20260924 03:14:51.359160 22664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:51.359742 22665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6015dc94dfe2414492bd7903325b5729. Latest consensus state: current_term: 1 leader_uuid: "6015dc94dfe2414492bd7903325b5729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6015dc94dfe2414492bd7903325b5729" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42933 } } }
I20260924 03:14:51.360062 22665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729 [sys.catalog]: This master's current role is: LEADER
W20260924 03:14:51.369809 22680 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:14:51.370056 22680 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:14:51.370205 22669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:51.371968 22669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:51.381232 22669 catalog_manager.cc:1347] Generated new cluster ID: 85c56cbb91a04c79b10681c2d40ad923
I20260924 03:14:51.381376 22669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:51.398059 22669 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:51.399565 22669 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:51.422900 22669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6015dc94dfe2414492bd7903325b5729: Generated new TSK 0
I20260924 03:14:51.423935 22669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:14:51.440867 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 22586
W20260924 03:14:51.442098 22648 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49077
W20260924 03:14:51.442857 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 22586: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:14:51.443042 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 22586
2026-09-24T03:14:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2014 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:51 dist-test-slave-5l98 krb5kdc[22685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:51 dist-test-slave-5l98 krb5kdc[22685](info): set up 2 sockets
Sep 24 03:14:51 dist-test-slave-5l98 krb5kdc[22685](info): commencing operation
krb5kdc: starting...
W20260924 03:14:52.581248 388 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:52 dist-test-slave-5l98 krb5kdc[22685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:14:52.857132 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44965
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:53.199585 22698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:53.200001 22698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:53.200187 22698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:53.208062 22698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:53.208273 22698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:53.208397 22698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:53.208527 22698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:53.208626 22698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:53.219525 22698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:53.221212 22698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:53.222643 22698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:53.235450 22706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:53.235401 22703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:53.235860 22704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:53.236882 22698 server_base.cc:1034] running on GCE node
I20260924 03:14:53.237841 22698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:53.239534 22698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:53.240914 22698 hybrid_clock.cc:648] HybridClock initialized: now 1790219693240861 us; error 39 us; skew 500 ppm
I20260924 03:14:53.241341 22698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:53 dist-test-slave-5l98 krb5kdc[22685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:53.244915 22698 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:14:53.246731 22698 webserver.cc:460] Webserver started at http://127.0.97.62:45965/ using document root <none> and password file <none>
I20260924 03:14:53.247445 22698 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:53.247555 22698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:53.247843 22698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:53.251122 22698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "952f3f0bf9ce470c85c604e49aa4c281"
format_stamp: "Formatted at 2026-09-24 03:14:53 on dist-test-slave-5l98"
server_key: "6205ad5588ecac27d8fe25e966ab5254"
server_key_iv: "d88e5998c38da483d63c0dde3460bc04"
server_key_version: "encryptionkey@0"
I20260924 03:14:53.251924 22698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "952f3f0bf9ce470c85c604e49aa4c281"
format_stamp: "Formatted at 2026-09-24 03:14:53 on dist-test-slave-5l98"
server_key: "6205ad5588ecac27d8fe25e966ab5254"
server_key_iv: "d88e5998c38da483d63c0dde3460bc04"
server_key_version: "encryptionkey@0"
I20260924 03:14:53.257781 22698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:14:53.261722 22713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:53.263454 22698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260924 03:14:53.263700 22698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "952f3f0bf9ce470c85c604e49aa4c281"
format_stamp: "Formatted at 2026-09-24 03:14:53 on dist-test-slave-5l98"
server_key: "6205ad5588ecac27d8fe25e966ab5254"
server_key_iv: "d88e5998c38da483d63c0dde3460bc04"
server_key_version: "encryptionkey@0"
I20260924 03:14:53.263931 22698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:53.270882 22698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:53.276130 22698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:53.276593 22698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:53.297848 22698 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44965
I20260924 03:14:53.298039 22764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44965 every 8 connection(s)
I20260924 03:14:53.299495 22698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:53.304533 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 22698
I20260924 03:14:53.304852 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:53.305418 22765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:53.305716 388 external_mini_cluster.cc:1428] Setting key 482f877fa2c6860df2d40fc34c81787e
I20260924 03:14:53.319728 22765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281: Bootstrap starting.
I20260924 03:14:53.321492 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:53.307672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38753 (local address 127.0.97.62:44965)
0924 03:14:53.308625 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:53.308640 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:53.308689 (+ 49us) server_negotiation.cc:407] Connection header received
0924 03:14:53.310764 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:53.310790 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:53.311197 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:53.311584 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:53.319452 (+ 7868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:53.319459 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:53.319475 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:53.319503 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:53.319578 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:53.319603 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:53.319931 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:53.320102 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 03:14:53.320397 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":404,"threads_started":1}
I20260924 03:14:53.323216 22765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:53.324482 22765 log.cc:824] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:53.325634 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:14:53.327579 22765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281: No bootstrap required, opened a new log
I20260924 03:14:53.333398 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:53.330710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43235 (local address 127.0.97.62:44965)
0924 03:14:53.330943 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:53.330954 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:53.331228 (+ 274us) server_negotiation.cc:407] Connection header received
0924 03:14:53.331398 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:53.331401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:53.331453 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:53.331542 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:53.332619 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:53.332623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:53.332627 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:14:53.332642 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:53.332685 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:14:53.332701 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:53.332875 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:53.333015 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 03:14:53.333064 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 03:14:53.336855 22765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952f3f0bf9ce470c85c604e49aa4c281" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44965 } }
I20260924 03:14:53.337227 22765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:14:53.337352 22765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 952f3f0bf9ce470c85c604e49aa4c281, State: Initialized, Role: FOLLOWER
I20260924 03:14:53.338071 22765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952f3f0bf9ce470c85c604e49aa4c281" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44965 } }
I20260924 03:14:53.338389 22765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:14:53.338548 22765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:14:53.338728 22765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:53.341076 22765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952f3f0bf9ce470c85c604e49aa4c281" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44965 } }
I20260924 03:14:53.341660 22765 leader_election.cc:302] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 952f3f0bf9ce470c85c604e49aa4c281; no voters:
I20260924 03:14:53.342316 22765 leader_election.cc:288] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:14:53.342484 22775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:53.343302 22775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281 [term 1 LEADER]: Becoming Leader. State: Replica: 952f3f0bf9ce470c85c604e49aa4c281, State: Running, Role: LEADER
I20260924 03:14:53.343986 22775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952f3f0bf9ce470c85c604e49aa4c281" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44965 } }
I20260924 03:14:53.344735 22765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:53.352240 22777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 952f3f0bf9ce470c85c604e49aa4c281. Latest consensus state: current_term: 1 leader_uuid: "952f3f0bf9ce470c85c604e49aa4c281" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952f3f0bf9ce470c85c604e49aa4c281" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44965 } } }
I20260924 03:14:53.352348 22776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "952f3f0bf9ce470c85c604e49aa4c281" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952f3f0bf9ce470c85c604e49aa4c281" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44965 } } }
I20260924 03:14:53.352711 22777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:53.352779 22776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:53.355044 22787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:53.360522 22787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:14:53.371192 22787 catalog_manager.cc:1347] Generated new cluster ID: b186c2218abf43ddb4031dae385cfb14
I20260924 03:14:53.371342 22787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:53.392079 22787 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:53.392700 22787 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:53.404955 22787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 952f3f0bf9ce470c85c604e49aa4c281: Generated new TSK 0
I20260924 03:14:53.406087 22787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:14:53.416559 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 22698
W20260924 03:14:53.417923 22760 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38753
W20260924 03:14:53.418695 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 22698: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:14:53.418874 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 22698
2026-09-24T03:14:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1975 ms)
[----------] 18 tests from AuthTokenIssuingTest (36435 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:53 dist-test-slave-5l98 krb5kdc[22796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:53 dist-test-slave-5l98 krb5kdc[22796](info): set up 2 sockets
Sep 24 03:14:53 dist-test-slave-5l98 krb5kdc[22796](info): commencing operation
krb5kdc: starting...
W20260924 03:14:54.549530 388 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.011s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:54 dist-test-slave-5l98 krb5kdc[22796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:14:54.833217 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46247
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46247,127.0.97.61:33565,127.0.97.60:35839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:55.173871 22809 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:14:55.174258 22809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:55.174399 22809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:55.174532 22809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:55.182204 22809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:55.182307 22809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:55.182407 22809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:55.182512 22809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:55.182595 22809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:55.193071 22809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46247,127.0.97.61:33565,127.0.97.60:35839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:55.194423 22809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:55.195765 22809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:55.207890 22817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:55.208086 22814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:55.208027 22815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:55.208839 22809 server_base.cc:1034] running on GCE node
I20260924 03:14:55.209726 22809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:55.211241 22809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:55.212569 22809 hybrid_clock.cc:648] HybridClock initialized: now 1790219695212446 us; error 57 us; skew 500 ppm
I20260924 03:14:55.212947 22809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:55 dist-test-slave-5l98 krb5kdc[22796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:55.216265 22809 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:14:55.217914 22809 webserver.cc:460] Webserver started at http://127.0.97.62:46121/ using document root <none> and password file <none>
I20260924 03:14:55.218690 22809 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:55.218844 22809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:55.219169 22809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:55.222362 22809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "4d8a16090b0746ed9764c32dfb447f74"
format_stamp: "Formatted at 2026-09-24 03:14:55 on dist-test-slave-5l98"
server_key: "7b70ff7d6a9ec4f15df84bbe1f056b18"
server_key_iv: "ece8bd4ab95642b9b22017e089097446"
server_key_version: "encryptionkey@0"
I20260924 03:14:55.223222 22809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "4d8a16090b0746ed9764c32dfb447f74"
format_stamp: "Formatted at 2026-09-24 03:14:55 on dist-test-slave-5l98"
server_key: "7b70ff7d6a9ec4f15df84bbe1f056b18"
server_key_iv: "ece8bd4ab95642b9b22017e089097446"
server_key_version: "encryptionkey@0"
I20260924 03:14:55.228796 22809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 03:14:55.232548 22824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:55.234062 22809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:14:55.234267 22809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "4d8a16090b0746ed9764c32dfb447f74"
format_stamp: "Formatted at 2026-09-24 03:14:55 on dist-test-slave-5l98"
server_key: "7b70ff7d6a9ec4f15df84bbe1f056b18"
server_key_iv: "ece8bd4ab95642b9b22017e089097446"
server_key_version: "encryptionkey@0"
I20260924 03:14:55.234542 22809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:55.249392 22809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:55.254616 22809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:55.254933 22809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:55.275717 22809 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46247
I20260924 03:14:55.275724 22875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46247 every 8 connection(s)
I20260924 03:14:55.277216 22809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:55.279850 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 22809
I20260924 03:14:55.280123 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:55.280855 388 external_mini_cluster.cc:1428] Setting key 515ad55740b4eedb77d26194352f4132
I20260924 03:14:55.281307 22876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:55.288813 22876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46247 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 24 03:14:55 dist-test-slave-5l98 krb5kdc[22796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:14:55.320173 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.293260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34506 (local address 127.0.97.62:46247)
0924 03:14:55.295270 (+ 2010us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:55.295276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:55.295321 (+ 45us) server_negotiation.cc:407] Connection header received
0924 03:14:55.297864 (+ 2543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:55.297871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:55.298148 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:55.298337 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:55.300075 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:55.301433 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:55.303913 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:55.304591 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:55.304750 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:55.312321 (+ 7571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:55.312331 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:55.312346 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:55.312380 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:55.315675 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:55.316546 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:55.316560 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:55.316562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:55.316720 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:55.317169 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:55.317177 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:55.317180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:55.317729 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:55.317944 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:55.318642 (+ 698us) server_negotiation.cc:299] Negotiation successful
0924 03:14:55.319023 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":1805,"thread_start_us":1552,"threads_started":1}
I20260924 03:14:55.320303 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.293260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46247 (local address 127.0.0.1:34506)
0924 03:14:55.294251 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:55.294303 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:55.297302 (+ 2999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:55.298426 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:55.298436 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:55.299304 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:55.299819 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:55.299846 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:55.301609 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:55.301618 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:55.303730 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:55.303753 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:55.304857 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:55.304870 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:55.305107 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:55.305844 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:14:55.305882 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:55.312022 (+ 6140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:55.315864 (+ 3842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:55.315878 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:55.315891 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:55.316350 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:55.316869 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:55.316874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:55.316877 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:55.317037 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:55.317881 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:55.317890 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:55.318158 (+ 268us) client_negotiation.cc:769] Sending connection context
0924 03:14:55.318480 (+ 322us) client_negotiation.cc:240] Negotiation successful
0924 03:14:55.319242 (+ 762us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":141,"spinlock_wait_cycles":16256,"thread_start_us":420,"threads_started":1}
I20260924 03:14:55.323355 22876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:14:55.324553 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.324010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33565 (local address 127.0.0.1:38938)
0924 03:14:55.324227 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:55.324362 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260924 03:14:55.324869 22825 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33565: connect: Connection refused (error 111)
I20260924 03:14:55.326608 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.326130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33565 (local address 127.0.0.1:38946)
0924 03:14:55.326321 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:55.326420 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 03:14:55.326839 22876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33565: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33565: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260924 03:14:55.348959 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33565
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46247,127.0.97.61:33565,127.0.97.60:35839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:14:55.376279 22876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } attempt: 1
I20260924 03:14:55.377642 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.377081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33565 (local address 127.0.0.1:38960)
0924 03:14:55.377335 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:55.377466 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 03:14:55.378800 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.378349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33565 (local address 127.0.0.1:38976)
0924 03:14:55.378506 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:55.378618 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 03:14:55.379006 22876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33565: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33565: connect: Connection refused (error 111)
I20260924 03:14:55.447312 22876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } attempt: 2
I20260924 03:14:55.448698 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.448062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33565 (local address 127.0.0.1:38978)
0924 03:14:55.448347 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:55.448467 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 03:14:55.449960 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.449489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33565 (local address 127.0.0.1:38982)
0924 03:14:55.449708 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:55.449792 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260924 03:14:55.450281 22876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33565: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33565: connect: Connection refused (error 111)
I20260924 03:14:55.541580 22876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } attempt: 3
I20260924 03:14:55.542809 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.542265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33565 (local address 127.0.0.1:38992)
0924 03:14:55.542513 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:55.542628 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 03:14:55.543915 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.543504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33565 (local address 127.0.0.1:39004)
0924 03:14:55.543662 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:55.543751 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 03:14:55.544147 22876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33565: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33565: connect: Connection refused (error 111)
I20260924 03:14:55.687444 22876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } attempt: 4
I20260924 03:14:55.688812 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.688157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33565 (local address 127.0.0.1:39016)
0924 03:14:55.688526 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:55.688641 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260924 03:14:55.689956 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.689530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33565 (local address 127.0.0.1:39028)
0924 03:14:55.689708 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:55.689804 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260924 03:14:55.690223 22876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33565: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33565: connect: Connection refused (error 111)
W20260924 03:14:55.692791 22883 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:14:55.693171 22883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:55.693254 22883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:55.693380 22883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:55.701079 22883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:55.701184 22883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:55.701275 22883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:55.701370 22883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:55.701478 22883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:55.711911 22883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46247,127.0.97.61:33565,127.0.97.60:35839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:55.713387 22883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:55.714730 22883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:55.727013 22889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:55.726966 22891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:55.727035 22888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:55.727967 22883 server_base.cc:1034] running on GCE node
I20260924 03:14:55.728856 22883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:55.730412 22883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:55.731685 22883 hybrid_clock.cc:648] HybridClock initialized: now 1790219695731679 us; error 129 us; skew 500 ppm
I20260924 03:14:55.732043 22883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:55 dist-test-slave-5l98 krb5kdc[22796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:55.735522 22883 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260924 03:14:55.737200 22883 webserver.cc:460] Webserver started at http://127.0.97.61:36433/ using document root <none> and password file <none>
I20260924 03:14:55.737982 22883 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:55.738108 22883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:55.738494 22883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:55.741851 22883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-1/data/instance:
uuid: "2cc84944c1e441d4b4ebd4d35f73bb15"
format_stamp: "Formatted at 2026-09-24 03:14:55 on dist-test-slave-5l98"
server_key: "97c34cd4b72c92527cc5e903e00bd61a"
server_key_iv: "fc5b1f12cfe3ab5fce3a75b569ae243e"
server_key_version: "encryptionkey@0"
I20260924 03:14:55.742702 22883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-1/wal/instance:
uuid: "2cc84944c1e441d4b4ebd4d35f73bb15"
format_stamp: "Formatted at 2026-09-24 03:14:55 on dist-test-slave-5l98"
server_key: "97c34cd4b72c92527cc5e903e00bd61a"
server_key_iv: "fc5b1f12cfe3ab5fce3a75b569ae243e"
server_key_version: "encryptionkey@0"
I20260924 03:14:55.748225 22883 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 03:14:55.752017 22898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:55.753685 22883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:14:55.753901 22883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-1/wal
uuid: "2cc84944c1e441d4b4ebd4d35f73bb15"
format_stamp: "Formatted at 2026-09-24 03:14:55 on dist-test-slave-5l98"
server_key: "97c34cd4b72c92527cc5e903e00bd61a"
server_key_iv: "fc5b1f12cfe3ab5fce3a75b569ae243e"
server_key_version: "encryptionkey@0"
I20260924 03:14:55.754153 22883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:55.769451 22883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:55.774556 22883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:55.774921 22883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:55.794529 22883 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:33565
I20260924 03:14:55.794550 22949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:33565 every 8 connection(s)
I20260924 03:14:55.796060 22883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-1/data/info.pb
I20260924 03:14:55.799984 22950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:55.806494 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 22883
I20260924 03:14:55.806728 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-1/wal/instance
I20260924 03:14:55.807178 388 external_mini_cluster.cc:1428] Setting key bde966fe9d06b87856efc329ca21fc30
I20260924 03:14:55.806736 22950 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46247 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:14:55 dist-test-slave-5l98 krb5kdc[22796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:14:55.830941 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.809632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34522 (local address 127.0.97.62:46247)
0924 03:14:55.809867 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:55.809873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:55.811206 (+ 1333us) server_negotiation.cc:407] Connection header received
0924 03:14:55.814457 (+ 3251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:55.814462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:55.814539 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:55.814634 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:55.817003 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:55.817704 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:55.819416 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:55.819823 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:55.819977 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:55.824060 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:55.824066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:55.824069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:55.824106 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:55.827115 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:55.828271 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:55.828276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:55.828278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:55.828412 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:55.829024 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:55.829028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:55.829030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:55.829335 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:55.829453 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:55.830505 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0924 03:14:55.830681 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260924 03:14:55.832520 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.810101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46247 (local address 127.0.0.1:34522)
0924 03:14:55.811005 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:55.811068 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:55.814069 (+ 3001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:55.814945 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:55.814959 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:55.815830 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:55.816776 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:55.816807 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:55.817890 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:55.817895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:55.819238 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:55.819251 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:55.820083 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:55.820090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:55.820333 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:55.821123 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:14:55.821148 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:55.823759 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:55.827304 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:55.827318 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:55.827341 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:55.828089 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:55.828580 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:55.828587 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:55.828593 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:55.828864 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:55.829493 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:55.829506 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:55.830070 (+ 564us) client_negotiation.cc:769] Sending connection context
0924 03:14:55.830407 (+ 337us) client_negotiation.cc:240] Negotiation successful
0924 03:14:55.831005 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":368,"threads_started":1}
I20260924 03:14:55.834340 22950 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 24 03:14:55 dist-test-slave-5l98 krb5kdc[22796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 03:14:55.851588 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.835569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39036 (local address 127.0.97.61:33565)
0924 03:14:55.836359 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:55.836370 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:55.836403 (+ 33us) server_negotiation.cc:407] Connection header received
0924 03:14:55.836462 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:55.836473 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:55.836707 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:55.836879 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:55.837929 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:55.839130 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:55.840966 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:55.841519 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:55.841615 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:55.845532 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:55.845545 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:55.845561 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:55.845589 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:55.848423 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:55.849311 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:55.849318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:55.849320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:55.849431 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:55.849886 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:55.849889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:55.849892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:55.850249 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:55.850406 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:55.850884 (+ 478us) server_negotiation.cc:299] Negotiation successful
0924 03:14:55.851273 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":155,"server-negotiator.queue_time_us":599,"thread_start_us":432,"threads_started":1}
I20260924 03:14:55.851598 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.835332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33565 (local address 127.0.0.1:39036)
0924 03:14:55.835539 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:55.835563 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:55.835675 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:55.836899 (+ 1224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:55.836906 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:55.837427 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:55.837654 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:55.837670 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:55.839284 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:55.839291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:55.840809 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:55.840827 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:55.841681 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:55.841689 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:55.841924 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:55.842592 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:14:55.842621 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:55.845291 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:55.848616 (+ 3325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:55.848625 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:55.848632 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:55.849168 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:55.849571 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:55.849579 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:55.849585 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:55.849764 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:55.850405 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:55.850412 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:55.850582 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 03:14:55.850764 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 03:14:55.851131 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 03:14:55.854499 22950 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:14:55.855880 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.855265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35839 (local address 127.0.0.1:49904)
0924 03:14:55.855484 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:55.855645 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260924 03:14:55.856150 22899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
I20260924 03:14:55.858161 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.857638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35839 (local address 127.0.0.1:49914)
0924 03:14:55.857860 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:55.857979 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 03:14:55.858458 22950 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35839: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260924 03:14:55.873414 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35839
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46247,127.0.97.61:33565,127.0.97.60:35839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:14:55.907903 22950 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } attempt: 1
I20260924 03:14:55.909247 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.908703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35839 (local address 127.0.0.1:49928)
0924 03:14:55.908927 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:55.909039 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 03:14:55.910460 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.910048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35839 (local address 127.0.0.1:49932)
0924 03:14:55.910209 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:55.910325 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 03:14:55.910723 22950 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35839: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
I20260924 03:14:55.979012 22950 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } attempt: 2
I20260924 03:14:55.980381 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.979795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35839 (local address 127.0.0.1:49942)
0924 03:14:55.980045 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:55.980165 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 03:14:55.981490 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.981091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35839 (local address 127.0.0.1:49946)
0924 03:14:55.981256 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:55.981343 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 03:14:55.981709 22950 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35839: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
I20260924 03:14:55.989512 22876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } attempt: 5
Sep 24 03:14:55 dist-test-slave-5l98 krb5kdc[22796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 03:14:56.004951 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.990348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33565 (local address 127.0.0.1:39050)
0924 03:14:55.990588 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:55.990605 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:55.990712 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:55.991193 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:55.991197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:55.991574 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:55.991748 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:55.991757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:55.993272 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:55.993278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:55.994331 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:55.994346 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:55.995453 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:55.995457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:55.995602 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:55.996307 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:14:55.996326 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:55.998485 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:56.002194 (+ 3709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:56.002198 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:56.002201 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:56.002586 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:56.003168 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:56.003180 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:56.003184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:56.003363 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:56.004087 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:56.004091 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:56.004246 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 03:14:56.004369 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 03:14:56.004684 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 03:14:56.005293 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:55.990478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39050 (local address 127.0.97.61:33565)
0924 03:14:55.990663 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:55.990670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:55.990695 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:14:55.990858 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:55.990864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:55.990964 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:55.991076 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:55.991943 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:55.993076 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:55.994513 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:55.995281 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:55.995417 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:55.998695 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:55.998702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:55.998706 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:55.998748 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:56.002038 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.002813 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.002819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.002823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.002975 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.003499 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.003504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.003507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.003943 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:56.004130 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:56.004558 (+ 428us) server_negotiation.cc:299] Negotiation successful
0924 03:14:56.004863 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:14:56.006310 22876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:14:56.007362 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.006973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35839 (local address 127.0.0.1:49948)
0924 03:14:56.007147 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:56.007225 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 03:14:56.008363 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.007958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35839 (local address 127.0.0.1:49960)
0924 03:14:56.008129 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:56.008242 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260924 03:14:56.008643 22876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35839: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
I20260924 03:14:56.059916 22876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } attempt: 1
I20260924 03:14:56.061373 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.060768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35839 (local address 127.0.0.1:49976)
0924 03:14:56.061001 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:56.061110 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 03:14:56.062438 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.062068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35839 (local address 127.0.0.1:49982)
0924 03:14:56.062223 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:56.062311 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 03:14:56.062687 22876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35839: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
I20260924 03:14:56.072960 22950 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } attempt: 3
I20260924 03:14:56.074178 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.073613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35839 (local address 127.0.0.1:49986)
0924 03:14:56.073866 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:56.073978 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 03:14:56.075436 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.074936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35839 (local address 127.0.0.1:49988)
0924 03:14:56.075177 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:56.075284 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260924 03:14:56.075711 22950 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35839: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
I20260924 03:14:56.130951 22876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } attempt: 2
I20260924 03:14:56.132194 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.131715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35839 (local address 127.0.0.1:50004)
0924 03:14:56.131946 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:56.132041 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 03:14:56.133309 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.132944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35839 (local address 127.0.0.1:50010)
0924 03:14:56.133105 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:56.133181 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 03:14:56.133529 22876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35839: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
I20260924 03:14:56.218957 22950 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } attempt: 4
I20260924 03:14:56.220204 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.219657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35839 (local address 127.0.0.1:45172)
0924 03:14:56.219909 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:56.220014 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 03:14:56.221423 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.221025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35839 (local address 127.0.0.1:45174)
0924 03:14:56.221195 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:56.221289 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 03:14:56.221743 22950 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35839: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
W20260924 03:14:56.226512 22957 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:14:56.226863 22957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:56.226930 22957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:56.227015 22957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:56.234683 22957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:56.234802 22957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:56.234862 22957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:56.234905 22957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:56.234967 22957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:56.239801 22876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } attempt: 3
I20260924 03:14:56.240933 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.240421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35839 (local address 127.0.0.1:45186)
0924 03:14:56.240690 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:56.240784 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260924 03:14:56.241971 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.241638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35839 (local address 127.0.0.1:45202)
0924 03:14:56.241802 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:56.241869 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 03:14:56.242223 22876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35839: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35839: connect: Connection refused (error 111)
I20260924 03:14:56.246080 22957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46247,127.0.97.61:33565,127.0.97.60:35839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:56.247560 22957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:56.249029 22957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:56.266553 22962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:56.266592 22965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:56.266500 22963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:56.267917 22957 server_base.cc:1034] running on GCE node
I20260924 03:14:56.269001 22957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:56.270560 22957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:56.271878 22957 hybrid_clock.cc:648] HybridClock initialized: now 1790219696271744 us; error 47 us; skew 500 ppm
I20260924 03:14:56.272286 22957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:56 dist-test-slave-5l98 krb5kdc[22796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:56.275916 22957 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260924 03:14:56.277887 22957 webserver.cc:460] Webserver started at http://127.0.97.60:33833/ using document root <none> and password file <none>
I20260924 03:14:56.278760 22957 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:56.278913 22957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:56.279278 22957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:56.282539 22957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-2/data/instance:
uuid: "190ab4514c72413794e8c738dc223978"
format_stamp: "Formatted at 2026-09-24 03:14:56 on dist-test-slave-5l98"
server_key: "e9eba7118819422b953375a7056609e8"
server_key_iv: "9000a14f067aabb25905e42597c3f538"
server_key_version: "encryptionkey@0"
I20260924 03:14:56.283442 22957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-2/wal/instance:
uuid: "190ab4514c72413794e8c738dc223978"
format_stamp: "Formatted at 2026-09-24 03:14:56 on dist-test-slave-5l98"
server_key: "e9eba7118819422b953375a7056609e8"
server_key_iv: "9000a14f067aabb25905e42597c3f538"
server_key_version: "encryptionkey@0"
I20260924 03:14:56.289152 22957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 03:14:56.292732 22972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:56.294312 22957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:14:56.294536 22957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-2/wal
uuid: "190ab4514c72413794e8c738dc223978"
format_stamp: "Formatted at 2026-09-24 03:14:56 on dist-test-slave-5l98"
server_key: "e9eba7118819422b953375a7056609e8"
server_key_iv: "9000a14f067aabb25905e42597c3f538"
server_key_version: "encryptionkey@0"
I20260924 03:14:56.294814 22957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:56.315570 22957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:56.320654 22957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:56.320943 22957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:56.340056 22957 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:35839
I20260924 03:14:56.340078 23023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:35839 every 8 connection(s)
I20260924 03:14:56.341530 22957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-2/data/info.pb
I20260924 03:14:56.345547 23024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:56.350220 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 22957
I20260924 03:14:56.350472 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219327163871-388-0/minicluster-data/master-2/wal/instance
I20260924 03:14:56.350921 388 external_mini_cluster.cc:1428] Setting key c3c18d3ba2336801bf195f8d2f4c23c2
I20260924 03:14:56.352955 23024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46247 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:14:56 dist-test-slave-5l98 krb5kdc[22796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 24 03:14:56 dist-test-slave-5l98 krb5kdc[22796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 03:14:56.375562 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.356294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56182 (local address 127.0.97.62:46247)
0924 03:14:56.356951 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:56.356960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:56.357916 (+ 956us) server_negotiation.cc:407] Connection header received
0924 03:14:56.358025 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:56.358033 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:56.358131 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:56.358274 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:56.360261 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.361270 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.363088 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.363688 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.363838 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:56.367998 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:56.368007 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:56.368013 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:56.368061 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:56.371245 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.372276 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.372282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.372286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.372470 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.373083 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.373087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.373089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.373755 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:56.373892 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:56.374806 (+ 914us) server_negotiation.cc:299] Negotiation successful
0924 03:14:56.375066 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":362,"threads_started":1}
I20260924 03:14:56.376688 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.356298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46247 (local address 127.0.0.1:56182)
0924 03:14:56.357543 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:56.357594 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:56.357807 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:56.358340 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:56.358346 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:56.359098 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:56.359913 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:56.359939 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.361446 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.361450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:56.362893 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:56.362909 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.364105 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.364110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:56.364386 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:56.365075 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:14:56.365104 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:56.367730 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:56.371430 (+ 3700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:56.371444 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:56.371459 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:56.372107 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:56.372640 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:56.372644 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:56.372646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:56.372928 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:56.373918 (+ 990us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:56.373926 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:56.374276 (+ 350us) client_negotiation.cc:769] Sending connection context
0924 03:14:56.374599 (+ 323us) client_negotiation.cc:240] Negotiation successful
0924 03:14:56.375137 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"mutex_wait_us":45,"spinlock_wait_cycles":20736,"thread_start_us":808,"threads_started":1}
I20260924 03:14:56.376708 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.353015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45216 (local address 127.0.97.60:35839)
0924 03:14:56.353870 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:56.353887 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:56.353945 (+ 58us) server_negotiation.cc:407] Connection header received
0924 03:14:56.356351 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:56.356377 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:56.356943 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:56.357429 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:56.359376 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.361167 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.362506 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.363312 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.363451 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:56.368394 (+ 4943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:56.368402 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:56.368417 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:56.368452 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:56.371743 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.372917 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.372926 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.372929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.373072 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.373566 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.373569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.373571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.374243 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:56.374560 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:56.374978 (+ 418us) server_negotiation.cc:299] Negotiation successful
0924 03:14:56.375230 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":594,"thread_start_us":349,"threads_started":1}
I20260924 03:14:56.378795 23024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:14:56 dist-test-slave-5l98 krb5kdc[22796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 24 03:14:56 dist-test-slave-5l98 krb5kdc[22796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:14:56.394943 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.379914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33565 (local address 127.0.0.1:46064)
0924 03:14:56.380230 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:56.380251 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:56.380376 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:56.380874 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:56.380879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:56.381267 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:56.381419 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:56.381427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.382443 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.382448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:56.383332 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:56.383342 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.384061 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.384066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:56.384263 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:56.384892 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:14:56.384910 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:56.387573 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:56.391797 (+ 4224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:56.391806 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:56.391812 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:56.392372 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:56.392992 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:56.392998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:56.393003 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:56.393196 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:56.393831 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:56.393837 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:56.394005 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 03:14:56.394173 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 03:14:56.394530 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260924 03:14:56.394972 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.380217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.0.97.61:33565)
0924 03:14:56.380398 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:56.380404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:56.380429 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:14:56.380545 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:56.380552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:56.380652 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:56.380754 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:56.381585 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.382285 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.383485 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.383906 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.384049 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:56.387884 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:56.387894 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:56.387902 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:56.387948 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:56.391512 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.392624 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.392631 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.392635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.392825 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.393336 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.393344 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.393347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.393682 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:56.393857 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:56.394287 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 03:14:56.394530 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:14:56.395342 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.380473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56196 (local address 127.0.97.62:46247)
0924 03:14:56.380699 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:56.380703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:56.380723 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:14:56.380769 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:56.380772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:56.380845 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:56.380961 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:56.382235 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.382920 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.384149 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.384660 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.384798 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:56.389285 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:56.389291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:56.389294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:56.389331 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:56.392581 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.393292 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.393299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.393303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.393434 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.393846 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.393851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.393854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.394134 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:56.394440 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:56.394756 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 03:14:56.394985 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260924 03:14:56.396656 23024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:14:56 dist-test-slave-5l98 krb5kdc[22796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 24 03:14:56 dist-test-slave-5l98 krb5kdc[22796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 03:14:56.414290 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.397519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35839 (local address 127.0.0.1:45232)
0924 03:14:56.397711 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:56.397729 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:56.397826 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:56.398187 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:56.398191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:56.398785 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:56.399016 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:56.399031 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.400681 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.400685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:56.402127 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:56.402145 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.403192 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.403197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:56.403440 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:56.404399 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:14:56.404428 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:56.407246 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:56.411500 (+ 4254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:56.411504 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:56.411508 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:56.412092 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:56.412587 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:56.412592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:56.412596 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:56.412767 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:56.413473 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:56.413477 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:56.413630 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 03:14:56.413765 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 03:14:56.414016 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 03:14:56.414587 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.397698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45232 (local address 127.0.97.60:35839)
0924 03:14:56.397862 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:56.397868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:56.397903 (+ 35us) server_negotiation.cc:407] Connection header received
0924 03:14:56.397952 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:56.397958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:56.398028 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:56.398114 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:56.399181 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.400516 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.402293 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.402966 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.403123 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:56.407531 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:56.407539 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:56.407544 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:56.407587 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:56.411298 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.412236 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.412244 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.412250 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.412408 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.412891 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.412896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.412900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.413296 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:56.413439 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:56.413895 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 03:14:56.414193 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":39}
I20260924 03:14:56.417047 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.397161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46080 (local address 127.0.97.61:33565)
0924 03:14:56.397326 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:56.397330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:56.397393 (+ 63us) server_negotiation.cc:407] Connection header received
0924 03:14:56.397652 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:56.397657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:56.397738 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:56.397842 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:56.399507 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.400707 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.402519 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.403325 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.403425 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:56.409664 (+ 6239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:56.409673 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:56.409678 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:56.409756 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:56.413684 (+ 3928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.414646 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.414652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.414655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.414825 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.415314 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.415319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.415322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.415883 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:56.416197 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:56.416474 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 03:14:56.416822 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:14:56.419605 22876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } attempt: 4
I20260924 03:14:56.421366 23024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978: Bootstrap starting.
I20260924 03:14:56.422675 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:14:56.425643 23024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:56.427012 23024 log.cc:824] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:14:56 dist-test-slave-5l98 krb5kdc[22796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 03:14:56.425491 22851 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4d8a16090b0746ed9764c32dfb447f74"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56196
I20260924 03:14:56.430323 23024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978: No bootstrap required, opened a new log
I20260924 03:14:56.437175 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.420316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35839 (local address 127.0.0.1:45242)
0924 03:14:56.420559 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:56.420580 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:56.420697 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:56.421152 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:56.421160 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:56.421822 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:56.421984 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:56.421993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.423360 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.423366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:56.424838 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:56.424854 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.425661 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.425666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:56.425901 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:56.426593 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:14:56.426611 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:56.429450 (+ 2839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:56.433900 (+ 4450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:56.433907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:56.433913 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:56.434579 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:56.435122 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:56.435127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:56.435130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:56.435322 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:56.436054 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:56.436060 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:56.436238 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 03:14:56.436378 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 03:14:56.436766 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 03:14:56.437233 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.420640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.0.97.60:35839)
0924 03:14:56.420832 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:56.420840 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:56.420862 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:14:56.420913 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:56.420917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:56.420991 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:56.421090 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:56.422141 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.423180 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.425000 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.425497 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.425658 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:56.429687 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:56.429694 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:56.429700 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:56.429736 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:56.433722 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.434768 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.434784 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.434787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.434978 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.435452 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.435457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.435461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.435921 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:56.436106 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:56.436587 (+ 481us) server_negotiation.cc:299] Negotiation successful
0924 03:14:56.436841 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 03:14:56.441514 23024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8a16090b0746ed9764c32dfb447f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46247 } } peers { permanent_uuid: "2cc84944c1e441d4b4ebd4d35f73bb15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } } peers { permanent_uuid: "190ab4514c72413794e8c738dc223978" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } }
I20260924 03:14:56.441998 23024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 190ab4514c72413794e8c738dc223978, State: Initialized, Role: FOLLOWER
I20260924 03:14:56.442544 23024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8a16090b0746ed9764c32dfb447f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46247 } } peers { permanent_uuid: "2cc84944c1e441d4b4ebd4d35f73bb15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } } peers { permanent_uuid: "190ab4514c72413794e8c738dc223978" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } }
I20260924 03:14:56.443423 22876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74: Bootstrap starting.
I20260924 03:14:56.443634 23031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8a16090b0746ed9764c32dfb447f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46247 } } peers { permanent_uuid: "2cc84944c1e441d4b4ebd4d35f73bb15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } } peers { permanent_uuid: "190ab4514c72413794e8c738dc223978" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } } }
I20260924 03:14:56.444031 23031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:14:56.444890 23024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:56.447978 22876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:56.449536 22876 log.cc:824] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:56.453132 22876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74: No bootstrap required, opened a new log
I20260924 03:14:56.460976 22876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8a16090b0746ed9764c32dfb447f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46247 } } peers { permanent_uuid: "2cc84944c1e441d4b4ebd4d35f73bb15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } } peers { permanent_uuid: "190ab4514c72413794e8c738dc223978" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } }
I20260924 03:14:56.461455 22876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d8a16090b0746ed9764c32dfb447f74, State: Initialized, Role: FOLLOWER
I20260924 03:14:56.461961 22876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8a16090b0746ed9764c32dfb447f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46247 } } peers { permanent_uuid: "2cc84944c1e441d4b4ebd4d35f73bb15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } } peers { permanent_uuid: "190ab4514c72413794e8c738dc223978" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } }
I20260924 03:14:56.463192 23045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8a16090b0746ed9764c32dfb447f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46247 } } peers { permanent_uuid: "2cc84944c1e441d4b4ebd4d35f73bb15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } } peers { permanent_uuid: "190ab4514c72413794e8c738dc223978" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } } }
I20260924 03:14:56.463879 23045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:14:56.464412 22876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:56.464576 22851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 03:14:56.464762 22851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [term 0 FOLLOWER]: Advancing to term 1
W20260924 03:14:56.465075 23043 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:14:56.465205 23043 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:14:56.454030 22957 master_runner.cc:186] Error getting master registration for 127.0.97.62:46247: OK, instance_id { permanent_uuid: "4d8a16090b0746ed9764c32dfb447f74" instance_seqno: 1790219695270148 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:14:56.465984 22957 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:14:56.468978 22851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8a16090b0746ed9764c32dfb447f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46247 } } peers { permanent_uuid: "2cc84944c1e441d4b4ebd4d35f73bb15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } } peers { permanent_uuid: "190ab4514c72413794e8c738dc223978" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } }
I20260924 03:14:56.471004 22851 leader_election.cc:288] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [CANDIDATE]: Term 1 election: Requested vote from peers 2cc84944c1e441d4b4ebd4d35f73bb15 (127.0.97.61:33565), 190ab4514c72413794e8c738dc223978 (127.0.97.60:35839)
I20260924 03:14:56.472710 22999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d8a16090b0746ed9764c32dfb447f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "190ab4514c72413794e8c738dc223978"
I20260924 03:14:56.473232 22999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:56.476936 22999 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d8a16090b0746ed9764c32dfb447f74 in term 1.
I20260924 03:14:56.477880 22825 leader_election.cc:302] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 190ab4514c72413794e8c738dc223978, 4d8a16090b0746ed9764c32dfb447f74; no voters:
I20260924 03:14:56.478478 23045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:14:56.480309 23045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [term 1 LEADER]: Becoming Leader. State: Replica: 4d8a16090b0746ed9764c32dfb447f74, State: Running, Role: LEADER
I20260924 03:14:56.500994 23045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8a16090b0746ed9764c32dfb447f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46247 } } peers { permanent_uuid: "2cc84944c1e441d4b4ebd4d35f73bb15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } } peers { permanent_uuid: "190ab4514c72413794e8c738dc223978" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } }
W20260924 03:14:56.508077 22830 tablet.cc:2369] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:14:56.512869 23050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d8a16090b0746ed9764c32dfb447f74. Latest consensus state: current_term: 1 leader_uuid: "4d8a16090b0746ed9764c32dfb447f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8a16090b0746ed9764c32dfb447f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46247 } } peers { permanent_uuid: "2cc84944c1e441d4b4ebd4d35f73bb15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } } peers { permanent_uuid: "190ab4514c72413794e8c738dc223978" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } } }
I20260924 03:14:56.513355 23050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:56.520956 22950 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } attempt: 5
I20260924 03:14:56.525280 23061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:14:56.541720 23061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 03:14:56 dist-test-slave-5l98 krb5kdc[22796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 03:14:56.563565 22999 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978 [term 1 FOLLOWER]: Refusing update from remote peer 4d8a16090b0746ed9764c32dfb447f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:14:56.564767 23050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "190ab4514c72413794e8c738dc223978" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:14:56.573287 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.521751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35839 (local address 127.0.0.1:45262)
0924 03:14:56.522148 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:56.522172 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:56.522302 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:56.548670 (+ 26368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:56.548677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:56.549209 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:56.549433 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:56.549449 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.551236 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.551244 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:56.552757 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:56.552778 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.554159 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.554168 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:56.554476 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:56.555375 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:14:56.555409 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:56.558441 (+ 3032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:56.563888 (+ 5447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:56.563894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:56.563897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:56.564514 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:56.564991 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:56.564998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:56.565001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:56.565184 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:56.571912 (+ 6728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:56.571922 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:56.572175 (+ 253us) client_negotiation.cc:769] Sending connection context
0924 03:14:56.572372 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 03:14:56.572789 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208}
W20260924 03:14:56.573764 22978 tablet.cc:2369] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:14:56.576670 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.523055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45262 (local address 127.0.97.60:35839)
0924 03:14:56.541381 (+ 18326us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:56.541388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:56.541415 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:14:56.541474 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:56.541480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:56.541561 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:56.549638 (+ 8077us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:56.549914 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.551034 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.553002 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.553926 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.558633 (+ 4707us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:56.558961 (+ 328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:56.558967 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:56.558971 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:56.559004 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:56.562642 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.564691 (+ 2049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.564696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.564699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.564853 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.565481 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.565488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.565491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.565855 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:56.575089 (+ 9234us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:56.575399 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 03:14:56.575811 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18171,"thread_start_us":18578,"threads_started":1}
I20260924 03:14:56.583918 22950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cc84944c1e441d4b4ebd4d35f73bb15: Bootstrap starting.
I20260924 03:14:56.593698 23031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d8a16090b0746ed9764c32dfb447f74. Latest consensus state: current_term: 1 leader_uuid: "4d8a16090b0746ed9764c32dfb447f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8a16090b0746ed9764c32dfb447f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46247 } } peers { permanent_uuid: "2cc84944c1e441d4b4ebd4d35f73bb15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } } peers { permanent_uuid: "190ab4514c72413794e8c738dc223978" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } } }
I20260924 03:14:56.594055 23031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:14:56.593878 22950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cc84944c1e441d4b4ebd4d35f73bb15: Neither blocks nor log segments found. Creating new log.
I20260924 03:14:56.599041 22950 log.cc:824] T 00000000000000000000000000000000 P 2cc84944c1e441d4b4ebd4d35f73bb15: Log is configured to *not* fsync() on all Append() calls
I20260924 03:14:56.602531 23065 mvcc.cc:205] Tried to move back new op lower bound from 7332739877112537088 to 7332739876874723328. Current Snapshot: MvccSnapshot[applied={T|T < 7332739877112537088}]
I20260924 03:14:56.612344 23067 mvcc.cc:205] Tried to move back new op lower bound from 7332739877112537088 to 7332739876874723328. Current Snapshot: MvccSnapshot[applied={T|T < 7332739877112537088}]
I20260924 03:14:56.616457 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.483971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46082 (local address 127.0.97.61:33565)
0924 03:14:56.484192 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:56.484201 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:56.525305 (+ 41104us) server_negotiation.cc:407] Connection header received
0924 03:14:56.526616 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:56.526627 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:56.526737 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:56.526830 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:56.539259 (+ 12429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.540766 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.556761 (+ 15995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.557687 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.557821 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:56.589322 (+ 31501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:56.589334 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:56.589342 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:56.589415 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:56.594304 (+ 4889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.607044 (+ 12740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.607054 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.607060 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.607392 (+ 332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.609514 (+ 2122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.609523 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.609529 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.610095 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:56.610443 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:56.615440 (+ 4997us) server_negotiation.cc:299] Negotiation successful
0924 03:14:56.615860 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 03:14:56.619380 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.483335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.0.97.62:46247)
0924 03:14:56.483571 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:56.483580 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:56.520400 (+ 36820us) server_negotiation.cc:407] Connection header received
0924 03:14:56.522397 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:56.522407 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:56.522522 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:56.522618 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:56.542493 (+ 19875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.544217 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.559355 (+ 15138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.560384 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.560540 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:56.594862 (+ 34322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:56.594873 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:56.594882 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:56.594945 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:56.601134 (+ 6189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.606152 (+ 5018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.606164 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.606170 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.606604 (+ 434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.610709 (+ 4105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.610719 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.610725 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.611701 (+ 976us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:56.612368 (+ 667us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:56.617704 (+ 5336us) server_negotiation.cc:299] Negotiation successful
0924 03:14:56.618369 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:14:56.622339 22950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cc84944c1e441d4b4ebd4d35f73bb15: No bootstrap required, opened a new log
I20260924 03:14:56.626123 23045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d8a16090b0746ed9764c32dfb447f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8a16090b0746ed9764c32dfb447f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46247 } } peers { permanent_uuid: "2cc84944c1e441d4b4ebd4d35f73bb15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } } peers { permanent_uuid: "190ab4514c72413794e8c738dc223978" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } } }
I20260924 03:14:56.626516 23045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:56.635784 23061 catalog_manager.cc:1347] Generated new cluster ID: b89bfe647c8e49c080c91aa405f3b250
I20260924 03:14:56.635905 23061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:14:56.642016 23074 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 03:14:56.642124 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.519750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45246 (local address 127.0.97.60:35839)
0924 03:14:56.519985 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:56.519995 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:56.537477 (+ 17482us) server_negotiation.cc:407] Connection header received
0924 03:14:56.538659 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:56.538670 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:56.538815 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:56.538911 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:56.562918 (+ 24007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.564397 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.582255 (+ 17858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.583249 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.583393 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:56.629361 (+ 45968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:56.629372 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:56.629380 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:56.629441 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:56.634363 (+ 4922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.636375 (+ 2012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.636384 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.636391 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.636619 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:56.637547 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:56.637556 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:56.637561 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:56.638446 (+ 885us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:56.639114 (+ 668us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:56.640477 (+ 1363us) server_negotiation.cc:299] Negotiation successful
0924 03:14:56.641169 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 03:14:56.646481 22809 master_runner.cc:186] Error getting master registration for 127.0.97.61:33565: OK, instance_id { permanent_uuid: "2cc84944c1e441d4b4ebd4d35f73bb15" instance_seqno: 1790219695789157 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:14:56.658463 22809 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:14:56.658423 22950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cc84944c1e441d4b4ebd4d35f73bb15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8a16090b0746ed9764c32dfb447f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46247 } } peers { permanent_uuid: "2cc84944c1e441d4b4ebd4d35f73bb15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } } peers { permanent_uuid: "190ab4514c72413794e8c738dc223978" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } }
I20260924 03:14:56.658969 22950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cc84944c1e441d4b4ebd4d35f73bb15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc84944c1e441d4b4ebd4d35f73bb15, State: Initialized, Role: FOLLOWER
I20260924 03:14:56.659590 22950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cc84944c1e441d4b4ebd4d35f73bb15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8a16090b0746ed9764c32dfb447f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46247 } } peers { permanent_uuid: "2cc84944c1e441d4b4ebd4d35f73bb15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } } peers { permanent_uuid: "190ab4514c72413794e8c738dc223978" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } }
I20260924 03:14:56.660282 23031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d8a16090b0746ed9764c32dfb447f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8a16090b0746ed9764c32dfb447f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46247 } } peers { permanent_uuid: "2cc84944c1e441d4b4ebd4d35f73bb15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } } peers { permanent_uuid: "190ab4514c72413794e8c738dc223978" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } } }
I20260924 03:14:56.660617 23031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:14:56.660694 23075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cc84944c1e441d4b4ebd4d35f73bb15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8a16090b0746ed9764c32dfb447f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46247 } } peers { permanent_uuid: "2cc84944c1e441d4b4ebd4d35f73bb15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } } peers { permanent_uuid: "190ab4514c72413794e8c738dc223978" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } } }
I20260924 03:14:56.661067 23075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cc84944c1e441d4b4ebd4d35f73bb15 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:14:56.663103 22950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cc84944c1e441d4b4ebd4d35f73bb15 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:14:56.663138 22925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d8a16090b0746ed9764c32dfb447f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2cc84944c1e441d4b4ebd4d35f73bb15"
I20260924 03:14:56.663482 22924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cc84944c1e441d4b4ebd4d35f73bb15 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:14:56.677803 22924 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2cc84944c1e441d4b4ebd4d35f73bb15 [term 1 FOLLOWER]: Refusing update from remote peer 4d8a16090b0746ed9764c32dfb447f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:14:56.685171 22925 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2cc84944c1e441d4b4ebd4d35f73bb15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d8a16090b0746ed9764c32dfb447f74 in term 1.
I20260924 03:14:56.686964 23045 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cc84944c1e441d4b4ebd4d35f73bb15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:14:56.688479 23061 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:14:56.690436 23061 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:14:56.715381 23061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74: Generated new TSK 0
I20260924 03:14:56.716610 23061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:14:56.747716 23081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cc84944c1e441d4b4ebd4d35f73bb15 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d8a16090b0746ed9764c32dfb447f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8a16090b0746ed9764c32dfb447f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46247 } } peers { permanent_uuid: "2cc84944c1e441d4b4ebd4d35f73bb15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } } peers { permanent_uuid: "190ab4514c72413794e8c738dc223978" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } } }
I20260924 03:14:56.748174 23081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cc84944c1e441d4b4ebd4d35f73bb15 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:14:56.748355 23045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d8a16090b0746ed9764c32dfb447f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8a16090b0746ed9764c32dfb447f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46247 } } peers { permanent_uuid: "2cc84944c1e441d4b4ebd4d35f73bb15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } } peers { permanent_uuid: "190ab4514c72413794e8c738dc223978" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } } }
I20260924 03:14:56.748680 23045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d8a16090b0746ed9764c32dfb447f74 [sys.catalog]: This master's current role is: LEADER
I20260924 03:14:56.748571 23075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cc84944c1e441d4b4ebd4d35f73bb15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d8a16090b0746ed9764c32dfb447f74. Latest consensus state: current_term: 1 leader_uuid: "4d8a16090b0746ed9764c32dfb447f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8a16090b0746ed9764c32dfb447f74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46247 } } peers { permanent_uuid: "2cc84944c1e441d4b4ebd4d35f73bb15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33565 } } peers { permanent_uuid: "190ab4514c72413794e8c738dc223978" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35839 } } }
I20260924 03:14:56.748864 23075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cc84944c1e441d4b4ebd4d35f73bb15 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:14:56.815779 23092 catalog_manager.cc:1259] Loaded cluster ID: b89bfe647c8e49c080c91aa405f3b250
I20260924 03:14:56.815922 23092 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2cc84944c1e441d4b4ebd4d35f73bb15: loading cluster ID for follower catalog manager: success
I20260924 03:14:56.819830 23092 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2cc84944c1e441d4b4ebd4d35f73bb15: acquiring CA information for follower catalog manager: success
I20260924 03:14:56.822431 23092 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2cc84944c1e441d4b4ebd4d35f73bb15: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 03:14:56.879331 23028 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:14:56.858223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56214 (local address 127.0.97.62:46247)
0924 03:14:56.858468 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:56.858474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:56.877133 (+ 18659us) server_negotiation.cc:407] Connection header received
0924 03:14:56.877471 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:56.877478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:56.877580 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:56.877673 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:56.879016 (+ 1343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56214: BlockingRecv error: recv got EOF from 127.0.0.1:56214 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260924 03:14:56.882071 22954 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:14:56.865082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46098 (local address 127.0.97.61:33565)
0924 03:14:56.865353 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:56.865360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:56.879980 (+ 14620us) server_negotiation.cc:407] Connection header received
0924 03:14:56.880263 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:56.880272 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:56.880410 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:56.880527 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:56.881762 (+ 1235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46098: BlockingRecv error: recv got EOF from 127.0.0.1:46098 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
W20260924 03:14:56.883821 23027 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:14:56.877493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45278 (local address 127.0.97.60:35839)
0924 03:14:56.877696 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:56.877703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:56.881484 (+ 3781us) server_negotiation.cc:407] Connection header received
0924 03:14:56.882147 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:56.882153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:56.882241 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:56.882321 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:56.883497 (+ 1176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45278: BlockingRecv error: recv got EOF from 127.0.0.1:45278 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 03:14:56.931393 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:56.912523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46114 (local address 127.0.97.61:33565)
0924 03:14:56.912707 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:56.912714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:56.923335 (+ 10621us) server_negotiation.cc:407] Connection header received
0924 03:14:56.923880 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:56.923887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:56.924141 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:56.924275 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:14:56.924994 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.926154 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.928297 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:56.928975 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:56.929080 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:56.929876 (+ 796us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:14:56.930564 (+ 688us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:14:56.930689 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:56.930930 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 03:14:56.931036 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 03:14:56.982014 23027 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:14:56.968506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45284 (local address 127.0.97.60:35839)
0924 03:14:56.968723 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:56.968729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:56.979412 (+ 10683us) server_negotiation.cc:407] Connection header received
0924 03:14:56.980012 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:56.980019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:56.980329 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:56.980441 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:56.981757 (+ 1316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45284: BlockingRecv error: recv got EOF from 127.0.0.1:45284 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:14:57.056171 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:57.018296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46118 (local address 127.0.97.61:33565)
0924 03:14:57.018507 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:57.018514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:57.048475 (+ 29961us) server_negotiation.cc:407] Connection header received
0924 03:14:57.049196 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:57.049203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:57.049298 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:57.049364 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:14:57.050091 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.051251 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.053426 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.054226 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.054360 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:57.055076 (+ 716us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:14:57.055324 (+ 248us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:14:57.055399 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:57.055646 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 03:14:57.055775 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 03:14:57.137475 23027 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:14:57.115525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.0.97.60:35839)
0924 03:14:57.115738 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:57.115747 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:57.135388 (+ 19641us) server_negotiation.cc:407] Connection header received
0924 03:14:57.135942 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:57.135948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:57.136061 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:57.136153 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:57.137232 (+ 1079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45286: BlockingRecv error: recv got EOF from 127.0.0.1:45286 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:14:57.171993 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:57.155339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46128 (local address 127.0.97.61:33565)
0924 03:14:57.155578 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:57.155588 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:57.164869 (+ 9281us) server_negotiation.cc:407] Connection header received
0924 03:14:57.165201 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:57.165209 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:57.165337 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:57.165407 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:14:57.166159 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.167308 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.169514 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.170294 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.170413 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:57.171078 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:14:57.171268 (+ 190us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:14:57.171332 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:57.171551 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 03:14:57.171659 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 03:14:57.218717 23027 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:14:57.210900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45294 (local address 127.0.97.60:35839)
0924 03:14:57.211075 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:57.211082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:57.217208 (+ 6126us) server_negotiation.cc:407] Connection header received
0924 03:14:57.217446 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:57.217452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:57.217538 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:57.217674 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:57.218484 (+ 810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45294: BlockingRecv error: recv got EOF from 127.0.0.1:45294 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:14:57.231398 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:57.225912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46134 (local address 127.0.97.61:33565)
0924 03:14:57.226169 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:57.226175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:57.226470 (+ 295us) server_negotiation.cc:407] Connection header received
0924 03:14:57.226701 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:57.226707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:57.226790 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:57.226898 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:14:57.227325 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.228116 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.229534 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.230035 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.230201 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:57.230608 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:14:57.230710 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:14:57.230821 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:57.230975 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 03:14:57.231061 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
W20260924 03:14:57.239746 23027 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:14:57.237782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45310 (local address 127.0.97.60:35839)
0924 03:14:57.237994 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:57.238000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:57.238220 (+ 220us) server_negotiation.cc:407] Connection header received
0924 03:14:57.238424 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:57.238429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:57.238509 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:57.238628 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:57.239491 (+ 863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45310: BlockingRecv error: recv got EOF from 127.0.0.1:45310 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 03:14:57.268253 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:57.261141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.0.97.61:33565)
0924 03:14:57.261380 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:57.261389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:57.261952 (+ 563us) server_negotiation.cc:407] Connection header received
0924 03:14:57.262191 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:57.262197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:57.262276 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:57.262389 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:14:57.262791 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.263506 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.265129 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.265601 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.265755 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:57.267464 (+ 1709us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:14:57.267591 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:14:57.267710 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:57.267836 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 03:14:57.267917 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 03:14:57.317926 23027 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:14:57.312366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45316 (local address 127.0.97.60:35839)
0924 03:14:57.312570 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:57.312576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:57.316018 (+ 3442us) server_negotiation.cc:407] Connection header received
0924 03:14:57.316261 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:57.316280 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:57.316414 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:57.316583 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:57.317705 (+ 1122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45316: BlockingRecv error: recv got EOF from 127.0.0.1:45316 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:14:57.346060 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:57.338523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46154 (local address 127.0.97.61:33565)
0924 03:14:57.338751 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:57.338760 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:57.339124 (+ 364us) server_negotiation.cc:407] Connection header received
0924 03:14:57.339565 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:57.339572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:57.339663 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:57.339728 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:14:57.340408 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.341635 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.343648 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.344343 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.344445 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:57.345182 (+ 737us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:14:57.345334 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:14:57.345402 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:57.345641 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 03:14:57.345746 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 03:14:57.381182 23027 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:14:57.378718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.0.97.60:35839)
0924 03:14:57.378901 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:57.378907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:57.379489 (+ 582us) server_negotiation.cc:407] Connection header received
0924 03:14:57.379698 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:57.379704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:57.379805 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:57.379877 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:57.380931 (+ 1054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45322: BlockingRecv error: recv got EOF from 127.0.0.1:45322 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 03:14:57.422420 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:57.417039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.0.97.61:33565)
0924 03:14:57.417273 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:57.417278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:57.417757 (+ 479us) server_negotiation.cc:407] Connection header received
0924 03:14:57.417916 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:57.417920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:57.417992 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:57.418080 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:14:57.418511 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.419233 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.420679 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.421200 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.421308 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:57.421692 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:14:57.421795 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:14:57.421869 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:57.422000 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 03:14:57.422066 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 03:14:57.429860 23027 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:14:57.428075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45334 (local address 127.0.97.60:35839)
0924 03:14:57.428356 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:57.428362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:57.428545 (+ 183us) server_negotiation.cc:407] Connection header received
0924 03:14:57.428731 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:57.428736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:57.428808 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:57.428893 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:57.429640 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45334: BlockingRecv error: recv got EOF from 127.0.0.1:45334 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260924 03:14:57.467129 23043 catalog_manager.cc:1259] Loaded cluster ID: b89bfe647c8e49c080c91aa405f3b250
I20260924 03:14:57.467231 23043 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978: loading cluster ID for follower catalog manager: success
I20260924 03:14:57.469797 23043 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978: acquiring CA information for follower catalog manager: success
I20260924 03:14:57.471653 23043 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 190ab4514c72413794e8c738dc223978: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:14:57.503086 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:57.497354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46174 (local address 127.0.97.61:33565)
0924 03:14:57.497606 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:57.497616 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:57.497994 (+ 378us) server_negotiation.cc:407] Connection header received
0924 03:14:57.498248 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:57.498253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:57.498378 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:57.498492 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:14:57.498929 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.499778 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.501276 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.501734 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.501909 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:57.502334 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:14:57.502452 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:14:57.502562 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:57.502719 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 03:14:57.502802 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:14:57.515331 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:57.509357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45338 (local address 127.0.97.60:35839)
0924 03:14:57.509742 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:57.509748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:57.509888 (+ 140us) server_negotiation.cc:407] Connection header received
0924 03:14:57.510124 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:57.510129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:57.510295 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:57.510451 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:14:57.510880 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.511594 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.513040 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.513419 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.513575 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:57.514070 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:14:57.514700 (+ 630us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:14:57.514860 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:57.514918 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 03:14:57.515004 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 03:14:57.529891 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:57.521879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46186 (local address 127.0.97.61:33565)
0924 03:14:57.522126 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:57.522135 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:57.523766 (+ 1631us) server_negotiation.cc:407] Connection header received
0924 03:14:57.523955 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:57.523960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:57.524048 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:57.524141 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:14:57.524841 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.525860 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.527908 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.528475 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.528627 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:57.529061 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:14:57.529177 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:14:57.529277 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:57.529486 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 03:14:57.529587 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 03:14:57.530424 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:57.521754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56218 (local address 127.0.97.62:46247)
0924 03:14:57.522633 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:57.522642 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:57.523416 (+ 774us) server_negotiation.cc:407] Connection header received
0924 03:14:57.523540 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:57.523544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:57.523734 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:57.523940 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:14:57.524659 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.525825 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.527478 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.528149 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.528292 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:57.528906 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:14:57.529387 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:14:57.529497 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:57.529592 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 03:14:57.529674 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":580,"threads_started":1}
I20260924 03:14:57.531539 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:57.523188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45348 (local address 127.0.97.60:35839)
0924 03:14:57.523374 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:57.523380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:57.525142 (+ 1762us) server_negotiation.cc:407] Connection header received
0924 03:14:57.525448 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:57.525452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:57.525517 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:57.525566 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:14:57.526300 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.527379 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.529324 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:57.529952 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:57.530075 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:57.530761 (+ 686us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:14:57.530877 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:14:57.530973 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:57.531147 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 03:14:57.531231 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 03:14:57.536006 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 22809
I20260924 03:14:57.537570 22871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:57.653198 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 22809
I20260924 03:14:57.674404 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 22883
I20260924 03:14:57.675614 22945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:57.785715 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 22883
I20260924 03:14:57.806331 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 22957
I20260924 03:14:57.807467 23019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:14:57.916656 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 22957
2026-09-24T03:14:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4502 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:14:58 dist-test-slave-5l98 krb5kdc[23196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:14:58 dist-test-slave-5l98 krb5kdc[23196](info): set up 2 sockets
Sep 24 03:14:58 dist-test-slave-5l98 krb5kdc[23196](info): commencing operation
krb5kdc: starting...
W20260924 03:14:59.057226 388 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/test-user.keytab.
Sep 24 03:14:59 dist-test-slave-5l98 krb5kdc[23196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:14:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:14:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 03:14:59.337373 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34675
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34675,127.0.97.61:38763,127.0.97.60:37871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:14:59.681351 23209 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:14:59.681737 23209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:14:59.681807 23209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:14:59.681896 23209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:14:59.689482 23209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:14:59.689617 23209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:14:59.689684 23209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:14:59.689730 23209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:14:59.689795 23209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:14:59.700554 23209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34675,127.0.97.61:38763,127.0.97.60:37871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:14:59.701879 23209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:14:59.703241 23209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:14:59.715610 23215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:59.715749 23214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:14:59.715726 23217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:14:59.716529 23209 server_base.cc:1034] running on GCE node
I20260924 03:14:59.717463 23209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:14:59.718948 23209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:14:59.720191 23209 hybrid_clock.cc:648] HybridClock initialized: now 1790219699720095 us; error 58 us; skew 500 ppm
I20260924 03:14:59.720675 23209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:14:59 dist-test-slave-5l98 krb5kdc[23196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:14:59.724216 23209 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 03:14:59.725977 23209 webserver.cc:460] Webserver started at http://127.0.97.62:36335/ using document root <none> and password file <none>
I20260924 03:14:59.726740 23209 fs_manager.cc:359] Metadata directory not provided
I20260924 03:14:59.726836 23209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:14:59.727193 23209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:14:59.730636 23209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-0/data/instance:
uuid: "0308f5c867b54d23a49cba5f2a8bb377"
format_stamp: "Formatted at 2026-09-24 03:14:59 on dist-test-slave-5l98"
server_key: "b69b8f78fec3cdca60d7d1a6c28aadbf"
server_key_iv: "47e4a962493f29f50d864754e175c61c"
server_key_version: "encryptionkey@0"
I20260924 03:14:59.731549 23209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-0/wal/instance:
uuid: "0308f5c867b54d23a49cba5f2a8bb377"
format_stamp: "Formatted at 2026-09-24 03:14:59 on dist-test-slave-5l98"
server_key: "b69b8f78fec3cdca60d7d1a6c28aadbf"
server_key_iv: "47e4a962493f29f50d864754e175c61c"
server_key_version: "encryptionkey@0"
I20260924 03:14:59.737304 23209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:14:59.741294 23224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:14:59.742997 23209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:14:59.743253 23209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-0/wal
uuid: "0308f5c867b54d23a49cba5f2a8bb377"
format_stamp: "Formatted at 2026-09-24 03:14:59 on dist-test-slave-5l98"
server_key: "b69b8f78fec3cdca60d7d1a6c28aadbf"
server_key_iv: "47e4a962493f29f50d864754e175c61c"
server_key_version: "encryptionkey@0"
I20260924 03:14:59.743512 23209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:14:59.778795 23209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:14:59.783758 23209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:14:59.784102 23209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:14:59.804394 23209 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34675
I20260924 03:14:59.804435 23275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34675 every 8 connection(s)
I20260924 03:14:59.805900 23209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-0/data/info.pb
I20260924 03:14:59.810220 23276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:14:59.814703 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 23209
I20260924 03:14:59.814917 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-0/wal/instance
I20260924 03:14:59.815387 388 external_mini_cluster.cc:1428] Setting key 9cb1a552d4e9e7e04afdfb8ce8a08795
I20260924 03:14:59.818020 23276 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:14:59 dist-test-slave-5l98 krb5kdc[23196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260924 03:14:59.845736 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:59.821740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38282 (local address 127.0.97.62:34675)
0924 03:14:59.823919 (+ 2179us) server_negotiation.cc:206] Beginning negotiation
0924 03:14:59.823926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:14:59.823974 (+ 48us) server_negotiation.cc:407] Connection header received
0924 03:14:59.825753 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:14:59.825764 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:14:59.826108 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:14:59.826402 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:14:59.827672 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:59.830056 (+ 2384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:59.831579 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:14:59.832617 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:14:59.832824 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:59.837077 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:14:59.837091 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:14:59.837111 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:14:59.837157 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:14:59.841136 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:59.841963 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:59.841971 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:59.841979 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:59.842121 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:14:59.842547 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:14:59.842551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:14:59.842554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:14:59.843059 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:14:59.843278 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 03:14:59.843837 (+ 559us) server_negotiation.cc:299] Negotiation successful
0924 03:14:59.844238 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":1980,"spinlock_wait_cycles":17280,"thread_start_us":1668,"threads_started":1}
I20260924 03:14:59.845788 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:59.821740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34675 (local address 127.0.0.1:38282)
0924 03:14:59.822740 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:59.822809 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 03:14:59.825347 (+ 2538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:14:59.826439 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:14:59.826445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:14:59.827064 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:14:59.827434 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:59.827451 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:59.830269 (+ 2818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:59.830273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:59.831406 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:14:59.831419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:14:59.832871 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:14:59.832877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:14:59.833012 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:14:59.833787 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:14:59.833818 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:14:59.836790 (+ 2972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:14:59.841290 (+ 4500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:59.841305 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:59.841321 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:59.841713 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:59.842245 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:14:59.842250 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:14:59.842254 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:14:59.842427 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:14:59.843250 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:14:59.843257 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:14:59.843456 (+ 199us) client_negotiation.cc:769] Sending connection context
0924 03:14:59.843705 (+ 249us) client_negotiation.cc:240] Negotiation successful
0924 03:14:59.844397 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":171,"thread_start_us":406,"threads_started":1}
I20260924 03:14:59.849140 23276 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:14:59.850370 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:59.849947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38763 (local address 127.0.0.1:51446)
0924 03:14:59.850128 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:59.850240 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 03:14:59.850646 23226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38763: connect: Connection refused (error 111)
I20260924 03:14:59.852296 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:59.851938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38763 (local address 127.0.0.1:51456)
0924 03:14:59.852085 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:59.852163 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 03:14:59.852566 23276 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38763: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38763: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260924 03:14:59.881842 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38763
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34675,127.0.97.61:38763,127.0.97.60:37871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:14:59.902002 23276 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } attempt: 1
I20260924 03:14:59.903635 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:59.902932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38763 (local address 127.0.0.1:51458)
0924 03:14:59.903271 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:59.903415 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 03:14:59.904907 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:59.904426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38763 (local address 127.0.0.1:51474)
0924 03:14:59.904627 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:59.904699 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260924 03:14:59.905099 23276 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38763: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38763: connect: Connection refused (error 111)
I20260924 03:14:59.973395 23276 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } attempt: 2
I20260924 03:14:59.974692 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:59.974181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38763 (local address 127.0.0.1:51490)
0924 03:14:59.974450 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:59.974548 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 03:14:59.975865 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:14:59.975481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38763 (local address 127.0.0.1:51498)
0924 03:14:59.975635 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 03:14:59.975732 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 03:14:59.976058 23276 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38763: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38763: connect: Connection refused (error 111)
I20260924 03:15:00.067364 23276 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } attempt: 3
I20260924 03:15:00.068747 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.068191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38763 (local address 127.0.0.1:51508)
0924 03:15:00.068426 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.068601 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 03:15:00.069993 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.069581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38763 (local address 127.0.0.1:51522)
0924 03:15:00.069773 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.069849 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260924 03:15:00.070178 23276 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38763: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38763: connect: Connection refused (error 111)
I20260924 03:15:00.213457 23276 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } attempt: 4
I20260924 03:15:00.214812 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.214226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38763 (local address 127.0.0.1:51534)
0924 03:15:00.214567 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.214666 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260924 03:15:00.215875 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.215517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38763 (local address 127.0.0.1:51536)
0924 03:15:00.215671 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.215759 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 03:15:00.216048 23276 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38763: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38763: connect: Connection refused (error 111)
W20260924 03:15:00.258430 23283 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:15:00.258869 23283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:00.258997 23283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:00.259141 23283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:00.266893 23283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:15:00.267040 23283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:00.267162 23283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:00.267259 23283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:15:00.267373 23283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:15:00.277932 23283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34675,127.0.97.61:38763,127.0.97.60:37871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:00.279464 23283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:00.281006 23283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:00.294265 23291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:00.294286 23288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:00.294308 23289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:00.295368 23283 server_base.cc:1034] running on GCE node
I20260924 03:15:00.296308 23283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:00.298007 23283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:00.299290 23283 hybrid_clock.cc:648] HybridClock initialized: now 1790219700299195 us; error 65 us; skew 500 ppm
I20260924 03:15:00.299700 23283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:00 dist-test-slave-5l98 krb5kdc[23196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:00.303447 23283 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260924 03:15:00.305357 23283 webserver.cc:460] Webserver started at http://127.0.97.61:41301/ using document root <none> and password file <none>
I20260924 03:15:00.306214 23283 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:00.306315 23283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:00.306602 23283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:00.309926 23283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-1/data/instance:
uuid: "2f41246ed4c04f04827ceda4328c54df"
format_stamp: "Formatted at 2026-09-24 03:15:00 on dist-test-slave-5l98"
server_key: "9c5ec7b74c5ea93d25f33f0725c1f2dd"
server_key_iv: "3476c2cd1aa8168d1e98bc7335065a61"
server_key_version: "encryptionkey@0"
I20260924 03:15:00.310817 23283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-1/wal/instance:
uuid: "2f41246ed4c04f04827ceda4328c54df"
format_stamp: "Formatted at 2026-09-24 03:15:00 on dist-test-slave-5l98"
server_key: "9c5ec7b74c5ea93d25f33f0725c1f2dd"
server_key_iv: "3476c2cd1aa8168d1e98bc7335065a61"
server_key_version: "encryptionkey@0"
I20260924 03:15:00.317042 23283 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260924 03:15:00.321053 23298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:00.322805 23283 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:15:00.323009 23283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-1/wal
uuid: "2f41246ed4c04f04827ceda4328c54df"
format_stamp: "Formatted at 2026-09-24 03:15:00 on dist-test-slave-5l98"
server_key: "9c5ec7b74c5ea93d25f33f0725c1f2dd"
server_key_iv: "3476c2cd1aa8168d1e98bc7335065a61"
server_key_version: "encryptionkey@0"
I20260924 03:15:00.323299 23283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:00.345836 23283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:00.350894 23283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:00.351209 23283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:00.371840 23283 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38763
I20260924 03:15:00.371943 23349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38763 every 8 connection(s)
I20260924 03:15:00.373385 23283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-1/data/info.pb
I20260924 03:15:00.374131 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 23283
I20260924 03:15:00.374351 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-1/wal/instance
I20260924 03:15:00.374969 388 external_mini_cluster.cc:1428] Setting key b674ed9d667483170fd9152d0febd8f7
I20260924 03:15:00.377586 23350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:00.384902 23350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 24 03:15:00 dist-test-slave-5l98 krb5kdc[23196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:15:00.412047 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.387783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38290 (local address 127.0.97.62:34675)
0924 03:15:00.388351 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:00.388356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:00.389446 (+ 1090us) server_negotiation.cc:407] Connection header received
0924 03:15:00.391511 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:00.391518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:00.391628 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:00.391756 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:00.393502 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.394231 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.395831 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.396317 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.396467 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:00.404388 (+ 7921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:00.404394 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:00.404398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:00.404458 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:00.407941 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:00.409146 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:00.409161 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:00.409165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:00.409309 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:00.409879 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:00.409891 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:00.409894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:00.410242 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:00.410449 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:00.411513 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0924 03:15:00.411715 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":316,"threads_started":1}
I20260924 03:15:00.413605 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.388361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34675 (local address 127.0.0.1:38290)
0924 03:15:00.389260 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.389306 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:00.391202 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:00.392004 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:00.392012 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:00.392670 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:00.393265 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:00.393284 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.394417 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.394421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:00.395665 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:00.395686 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.396658 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.396692 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:00.397040 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:00.397923 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:00.397946 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:00.403865 (+ 5919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:00.408281 (+ 4416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:00.408298 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:00.408316 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:00.408965 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:00.409453 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:00.409458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:00.409467 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:00.409721 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:00.410401 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:00.410410 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:00.410976 (+ 566us) client_negotiation.cc:769] Sending connection context
0924 03:15:00.411353 (+ 377us) client_negotiation.cc:240] Negotiation successful
0924 03:15:00.412191 (+ 838us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":340,"threads_started":1}
I20260924 03:15:00.415800 23350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:15:00 dist-test-slave-5l98 krb5kdc[23196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 03:15:00.435665 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.416773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38763 (local address 127.0.0.1:51552)
0924 03:15:00.417074 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.417098 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:00.417230 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:00.418359 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:00.418367 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:00.418951 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:00.419193 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:00.419209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.421078 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.421087 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:00.422738 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:00.422758 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.424439 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.424448 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:00.424724 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:00.425511 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:00.425541 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:00.428553 (+ 3012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:00.432967 (+ 4414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:00.432972 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:00.432975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:00.433421 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:00.433827 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:00.433831 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:00.433834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:00.434065 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:00.434804 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:00.434808 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:00.434998 (+ 190us) client_negotiation.cc:769] Sending connection context
0924 03:15:00.435134 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 03:15:00.435383 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 03:15:00.435943 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.417079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51552 (local address 127.0.97.61:38763)
0924 03:15:00.417821 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:00.417828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:00.417858 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:15:00.417909 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:00.417915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:00.418122 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:00.418252 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:00.419626 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.420884 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.422929 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.423999 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.424153 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:00.428857 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:00.428903 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:00.428924 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:00.428973 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:00.432790 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:00.433553 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:00.433560 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:00.433562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:00.433689 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:00.434206 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:00.434210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:00.434212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:00.434599 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:00.434786 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:00.435319 (+ 533us) server_negotiation.cc:299] Negotiation successful
0924 03:15:00.435532 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":427,"threads_started":1}
I20260924 03:15:00.439069 23350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:15:00.440351 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.439730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37871 (local address 127.0.0.1:37594)
0924 03:15:00.439954 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.440140 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260924 03:15:00.440688 23299 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
I20260924 03:15:00.442764 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.442150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37871 (local address 127.0.0.1:37610)
0924 03:15:00.442520 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.442625 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256}
W20260924 03:15:00.443089 23350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37871: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260924 03:15:00.445089 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
/tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37871
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34675,127.0.97.61:38763,127.0.97.60:37871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:15:00.492477 23350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } attempt: 1
I20260924 03:15:00.493860 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.493344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37871 (local address 127.0.0.1:37626)
0924 03:15:00.493590 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.493684 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 03:15:00.494884 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.494524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37871 (local address 127.0.0.1:37636)
0924 03:15:00.494672 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.494777 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 03:15:00.495122 23350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37871: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
I20260924 03:15:00.515350 23276 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } attempt: 5
Sep 24 03:15:00 dist-test-slave-5l98 krb5kdc[23196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 03:15:00.531404 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.516267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38763 (local address 127.0.0.1:51558)
0924 03:15:00.516520 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.516549 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:00.516689 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:00.517159 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:00.517163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:00.517558 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:00.517772 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:00.517783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.519339 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.519343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:00.520185 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:00.520195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.521243 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.521247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:00.521440 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:00.522251 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:00.522269 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:00.524324 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:00.528382 (+ 4058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:00.528386 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:00.528389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:00.529102 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:00.529597 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:00.529600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:00.529602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:00.529786 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:00.530567 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:00.530571 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:00.530768 (+ 197us) client_negotiation.cc:769] Sending connection context
0924 03:15:00.530894 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 03:15:00.531141 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 03:15:00.531762 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.516416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51558 (local address 127.0.97.61:38763)
0924 03:15:00.516706 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:00.516712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:00.516742 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:15:00.516813 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:00.516819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:00.516924 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:00.517042 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:00.517954 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.519153 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.520357 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.521042 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.521188 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:00.524582 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:00.524589 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:00.524594 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:00.524638 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:00.528232 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:00.529267 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:00.529276 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:00.529282 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:00.529461 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:00.529917 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:00.529925 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:00.529929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:00.530425 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:00.530608 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:00.531064 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 03:15:00.531378 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260924 03:15:00.532917 23276 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:15:00.534013 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.533603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37871 (local address 127.0.0.1:37640)
0924 03:15:00.533790 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.533866 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 03:15:00.535080 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.534750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37871 (local address 127.0.0.1:37644)
0924 03:15:00.534916 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.534989 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 03:15:00.535343 23276 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37871: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
I20260924 03:15:00.563393 23350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } attempt: 2
I20260924 03:15:00.564985 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.564403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37871 (local address 127.0.0.1:37648)
0924 03:15:00.564658 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.564823 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260924 03:15:00.566020 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.565677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37871 (local address 127.0.0.1:37662)
0924 03:15:00.565826 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.565908 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 03:15:00.566214 23350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37871: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
I20260924 03:15:00.586709 23276 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } attempt: 1
I20260924 03:15:00.588001 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.587501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37871 (local address 127.0.0.1:37668)
0924 03:15:00.587730 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.587835 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 03:15:00.589208 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.588839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37871 (local address 127.0.0.1:37674)
0924 03:15:00.588991 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.589103 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 03:15:00.589416 23276 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37871: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
I20260924 03:15:00.657459 23350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } attempt: 3
I20260924 03:15:00.657640 23276 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } attempt: 2
I20260924 03:15:00.658903 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.658281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37871 (local address 127.0.0.1:37678)
0924 03:15:00.658622 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.658717 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211}
I20260924 03:15:00.658903 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.658308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37871 (local address 127.0.0.1:37682)
0924 03:15:00.658533 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.658642 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 03:15:00.659982 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.659621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37871 (local address 127.0.0.1:37692)
0924 03:15:00.659803 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.659890 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 03:15:00.660173 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.659628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37871 (local address 127.0.0.1:37706)
0924 03:15:00.659917 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.660012 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
W20260924 03:15:00.660213 23276 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37871: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
W20260924 03:15:00.660470 23350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37871: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
I20260924 03:15:00.766481 23276 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } attempt: 3
I20260924 03:15:00.767822 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.767203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37871 (local address 127.0.0.1:37718)
0924 03:15:00.767472 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.767597 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 03:15:00.769176 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.768602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37871 (local address 127.0.0.1:37728)
0924 03:15:00.768893 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.769003 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260924 03:15:00.769423 23276 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37871: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
W20260924 03:15:00.798147 23358 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:15:00.798516 23358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:00.798686 23358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:00.798849 23358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 03:15:00.803766 23350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } attempt: 4
I20260924 03:15:00.805105 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.804568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37871 (local address 127.0.0.1:37744)
0924 03:15:00.804812 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.804916 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 03:15:00.806174 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.805812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37871 (local address 127.0.0.1:37748)
0924 03:15:00.805964 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.806046 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 03:15:00.806407 23350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37871: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37871: connect: Connection refused (error 111)
W20260924 03:15:00.807389 23358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:15:00.807463 23358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:00.807586 23358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:00.807670 23358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:15:00.807763 23358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:15:00.818351 23358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34675,127.0.97.61:38763,127.0.97.60:37871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:00.819705 23358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:00.821209 23358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:00.833716 23363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:00.833788 23366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:00.833741 23364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:00.834502 23358 server_base.cc:1034] running on GCE node
I20260924 03:15:00.835299 23358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:00.836956 23358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:00.838236 23358 hybrid_clock.cc:648] HybridClock initialized: now 1790219700838155 us; error 56 us; skew 500 ppm
I20260924 03:15:00.838636 23358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:00 dist-test-slave-5l98 krb5kdc[23196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:00.842118 23358 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260924 03:15:00.843878 23358 webserver.cc:460] Webserver started at http://127.0.97.60:43657/ using document root <none> and password file <none>
I20260924 03:15:00.844694 23358 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:00.844847 23358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:00.845193 23358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:00.848371 23358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-2/data/instance:
uuid: "6a389d02268e457b8da60f9bf1329ebe"
format_stamp: "Formatted at 2026-09-24 03:15:00 on dist-test-slave-5l98"
server_key: "952c686a68ace044feec60a3c9c53194"
server_key_iv: "b72a5f3b01ff2bcb6fd0c86959c4402b"
server_key_version: "encryptionkey@0"
I20260924 03:15:00.849278 23358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-2/wal/instance:
uuid: "6a389d02268e457b8da60f9bf1329ebe"
format_stamp: "Formatted at 2026-09-24 03:15:00 on dist-test-slave-5l98"
server_key: "952c686a68ace044feec60a3c9c53194"
server_key_iv: "b72a5f3b01ff2bcb6fd0c86959c4402b"
server_key_version: "encryptionkey@0"
I20260924 03:15:00.854683 23358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 03:15:00.858629 23373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:00.860344 23358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:15:00.860675 23358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-2/wal
uuid: "6a389d02268e457b8da60f9bf1329ebe"
format_stamp: "Formatted at 2026-09-24 03:15:00 on dist-test-slave-5l98"
server_key: "952c686a68ace044feec60a3c9c53194"
server_key_iv: "b72a5f3b01ff2bcb6fd0c86959c4402b"
server_key_version: "encryptionkey@0"
I20260924 03:15:00.860952 23358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:00.883879 23358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:00.888299 23358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:00.888729 23358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:00.908812 23358 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:37871
I20260924 03:15:00.908854 23424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:37871 every 8 connection(s)
I20260924 03:15:00.910260 23358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-2/data/info.pb
I20260924 03:15:00.912837 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu as pid 23358
I20260924 03:15:00.913022 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGuLdJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219327163871-388-0/minicluster-data/master-2/wal/instance
I20260924 03:15:00.913563 388 external_mini_cluster.cc:1428] Setting key bf0642404286ca6ed4c64a89e3ef1bbe
I20260924 03:15:00.915053 23425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:00.924971 23425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:15:00 dist-test-slave-5l98 krb5kdc[23196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 24 03:15:00 dist-test-slave-5l98 krb5kdc[23196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 03:15:00.939394 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.915528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.0.97.60:37871)
0924 03:15:00.916364 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:00.916375 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:00.916416 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:15:00.918147 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:00.918169 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:00.918579 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:00.918939 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:00.920983 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.922470 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.924349 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.925112 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.925307 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:00.930634 (+ 5327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:00.930644 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:00.930657 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:00.930701 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:00.934764 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:00.935679 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:00.935689 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:00.935700 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:00.935914 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:00.936352 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:00.936358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:00.936362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:00.936995 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:00.937227 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:00.937762 (+ 535us) server_negotiation.cc:299] Negotiation successful
0924 03:15:00.938241 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":386,"threads_started":1}
I20260924 03:15:00.946163 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.928462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34675 (local address 127.0.0.1:38296)
0924 03:15:00.929454 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.929496 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:00.929684 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:00.930115 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:00.930120 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:00.931092 (+ 972us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:00.931497 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:00.931508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.932686 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.932692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:00.934528 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:00.934544 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.935739 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.935746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:00.935945 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:00.936799 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:00.936817 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:00.939296 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:00.943355 (+ 4059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:00.943368 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:00.943376 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:00.943840 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:00.944255 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:00.944260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:00.944263 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:00.944399 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:00.944960 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:00.944968 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:00.945170 (+ 202us) client_negotiation.cc:769] Sending connection context
0924 03:15:00.945364 (+ 194us) client_negotiation.cc:240] Negotiation successful
0924 03:15:00.945698 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":556,"threads_started":1}
I20260924 03:15:00.946151 23429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.928464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.0.97.62:34675)
0924 03:15:00.929043 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:00.929051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:00.929621 (+ 570us) server_negotiation.cc:407] Connection header received
0924 03:15:00.929829 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:00.929838 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:00.929935 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:00.930061 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:00.931671 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.932483 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.934844 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.935506 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.935669 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:00.939521 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:00.939528 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:00.939532 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:00.939585 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:00.943190 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:00.943974 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:00.943981 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:00.943984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:00.944125 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:00.944543 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:00.944548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:00.944553 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:00.944826 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:00.945010 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:00.945473 (+ 463us) server_negotiation.cc:299] Negotiation successful
0924 03:15:00.945697 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":270,"threads_started":1}
I20260924 03:15:00.946643 23276 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } attempt: 4
I20260924 03:15:00.948702 23425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:15:00 dist-test-slave-5l98 krb5kdc[23196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 24 03:15:00 dist-test-slave-5l98 krb5kdc[23196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 24 03:15:00 dist-test-slave-5l98 krb5kdc[23196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 03:15:00.961299 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.942698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.0.97.62:34675)
0924 03:15:00.943451 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:00.943455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:00.943475 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:15:00.943518 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:00.943521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:00.943576 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:00.943704 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:00.945019 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.945847 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.948120 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.948846 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.948966 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:00.953874 (+ 4908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:00.953880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:00.953883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:00.953915 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:00.957655 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:00.958421 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:00.958427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:00.958430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:00.958620 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:00.959065 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:00.959071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:00.959075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:00.959434 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:00.959718 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:00.960689 (+ 971us) server_negotiation.cc:299] Negotiation successful
0924 03:15:00.960919 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":419,"threads_started":1}
I20260924 03:15:00.963225 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.947307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37871 (local address 127.0.0.1:37762)
0924 03:15:00.947666 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.947684 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:00.947773 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:00.948084 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:00.948089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:00.948706 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:00.948887 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:00.948902 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.950324 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.950330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:00.951248 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:00.951260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.952310 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.952317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:00.952551 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:00.953196 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:00.953220 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:00.956356 (+ 3136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:00.960269 (+ 3913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:00.960277 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:00.960281 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:00.960783 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:00.961227 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:00.961234 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:00.961243 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:00.961487 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:00.962223 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:00.962227 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:00.962353 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 03:15:00.962499 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 03:15:00.962862 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208}
I20260924 03:15:00.964389 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.947394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.0.97.60:37871)
0924 03:15:00.947583 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:00.947590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:00.947750 (+ 160us) server_negotiation.cc:407] Connection header received
0924 03:15:00.947854 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:00.947859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:00.947935 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:00.948040 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:00.949039 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.950172 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.951407 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.952115 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.952270 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:00.956627 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:00.956637 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:00.956644 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:00.956691 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:00.960127 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:00.960916 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:00.960922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:00.960926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:00.961089 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:00.961628 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:00.961632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:00.961635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:00.961984 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:00.962141 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:00.963795 (+ 1654us) server_negotiation.cc:299] Negotiation successful
0924 03:15:00.964056 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:15:00.970752 23276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377: Bootstrap starting.
I20260924 03:15:00.970996 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.949581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38763 (local address 127.0.0.1:51560)
0924 03:15:00.949804 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.949821 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:00.949924 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:00.951452 (+ 1528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:00.951457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:00.951967 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:00.952195 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:00.952209 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.953889 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.953895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:00.955168 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:00.955182 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.956225 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.956231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:00.956416 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:00.957074 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:00.957104 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:00.960229 (+ 3125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:00.964911 (+ 4682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:00.964916 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:00.964922 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:00.965513 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:00.968850 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:00.968855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:00.968860 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:00.969076 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:00.969914 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:00.969919 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:00.970119 (+ 200us) client_negotiation.cc:769] Sending connection context
0924 03:15:00.970303 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 03:15:00.970628 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"spinlock_wait_cycles":32640}
I20260924 03:15:00.971397 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.949575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.0.97.61:38763)
0924 03:15:00.949776 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:00.949781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:00.949868 (+ 87us) server_negotiation.cc:407] Connection header received
0924 03:15:00.950069 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:00.950075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:00.950159 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:00.950273 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:00.952367 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.953691 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.955344 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.956041 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.956208 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:00.960462 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:00.960470 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:00.960476 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:00.960547 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:00.964719 (+ 4172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:00.965672 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:00.965677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:00.965680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:00.968623 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:00.969225 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:00.969231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:00.969234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:00.969745 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:00.969910 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:00.970408 (+ 498us) server_negotiation.cc:299] Negotiation successful
0924 03:15:00.970865 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"spinlock_wait_cycles":27392}
I20260924 03:15:00.972653 23425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:15:00.974752 23276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377: Neither blocks nor log segments found. Creating new log.
Sep 24 03:15:00 dist-test-slave-5l98 krb5kdc[23196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 03:15:00.976255 23276 log.cc:824] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:00.980038 23276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377: No bootstrap required, opened a new log
Sep 24 03:15:00 dist-test-slave-5l98 krb5kdc[23196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 03:15:00.984330 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.963469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51568 (local address 127.0.97.61:38763)
0924 03:15:00.964087 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:00.964093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:00.964123 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:15:00.964184 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:00.964189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:00.964268 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:00.964381 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:00.966031 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.966883 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.968734 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.969398 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.969565 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:00.976857 (+ 7292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:00.976865 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:00.976870 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:00.976919 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:00.980461 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:00.981561 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:00.981567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:00.981571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:00.981822 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:00.982322 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:00.982327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:00.982331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:00.982715 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:00.983023 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:00.983409 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 03:15:00.983677 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":263,"threads_started":1}
I20260924 03:15:00.989209 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.973434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37871 (local address 127.0.0.1:37774)
0924 03:15:00.974030 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:00.974049 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:00.974169 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:00.974680 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:00.974683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:00.975067 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:00.975206 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:00.975214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.976265 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.976273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:00.977621 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:00.977639 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.978663 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.978670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:00.978889 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:00.979435 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:00.979497 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:00.982485 (+ 2988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:00.986338 (+ 3853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:00.986343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:00.986347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:00.986751 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:00.987185 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:00.987190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:00.987194 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:00.987367 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:00.988054 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:00.988058 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:00.988212 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 03:15:00.988394 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 03:15:00.988802 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494}
I20260924 03:15:00.989419 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:00.973693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37774 (local address 127.0.97.60:37871)
0924 03:15:00.973908 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:00.973915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:00.974270 (+ 355us) server_negotiation.cc:407] Connection header received
0924 03:15:00.974319 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:00.974325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:00.974400 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:00.974490 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:00.975355 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.976126 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.977782 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:00.978457 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:00.978590 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:00.982699 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:00.982704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:00.982707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:00.982738 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:00.986187 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:00.986872 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:00.986880 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:00.986883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:00.987051 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:00.987495 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:00.987501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:00.987503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:00.987924 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:00.988091 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:00.988592 (+ 501us) server_negotiation.cc:299] Negotiation successful
0924 03:15:00.988972 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":132,"server-negotiator.queue_time_us":56}
I20260924 03:15:00.989957 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:15:00.992628 23276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0308f5c867b54d23a49cba5f2a8bb377" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } } peers { permanent_uuid: "2f41246ed4c04f04827ceda4328c54df" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } } peers { permanent_uuid: "6a389d02268e457b8da60f9bf1329ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } }
I20260924 03:15:00.993112 23276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0308f5c867b54d23a49cba5f2a8bb377, State: Initialized, Role: FOLLOWER
I20260924 03:15:00.993158 23251 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0308f5c867b54d23a49cba5f2a8bb377"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38312
I20260924 03:15:00.993630 23276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0308f5c867b54d23a49cba5f2a8bb377" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } } peers { permanent_uuid: "2f41246ed4c04f04827ceda4328c54df" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } } peers { permanent_uuid: "6a389d02268e457b8da60f9bf1329ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } }
I20260924 03:15:00.994531 23425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a389d02268e457b8da60f9bf1329ebe: Bootstrap starting.
I20260924 03:15:00.994607 23434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0308f5c867b54d23a49cba5f2a8bb377" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } } peers { permanent_uuid: "2f41246ed4c04f04827ceda4328c54df" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } } peers { permanent_uuid: "6a389d02268e457b8da60f9bf1329ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } } }
I20260924 03:15:00.995042 23434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:15:00.995930 23276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:15:00.995985 23251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 03:15:00.996166 23251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:00.998224 23425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a389d02268e457b8da60f9bf1329ebe: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:00.999315 23425 log.cc:824] T 00000000000000000000000000000000 P 6a389d02268e457b8da60f9bf1329ebe: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:01.000824 23251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0308f5c867b54d23a49cba5f2a8bb377" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } } peers { permanent_uuid: "2f41246ed4c04f04827ceda4328c54df" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } } peers { permanent_uuid: "6a389d02268e457b8da60f9bf1329ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } }
I20260924 03:15:01.002975 23425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a389d02268e457b8da60f9bf1329ebe: No bootstrap required, opened a new log
I20260924 03:15:01.003548 23251 leader_election.cc:288] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [CANDIDATE]: Term 1 election: Requested vote from peers 2f41246ed4c04f04827ceda4328c54df (127.0.97.61:38763), 6a389d02268e457b8da60f9bf1329ebe (127.0.97.60:37871)
I20260924 03:15:01.020026 23425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a389d02268e457b8da60f9bf1329ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0308f5c867b54d23a49cba5f2a8bb377" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } } peers { permanent_uuid: "2f41246ed4c04f04827ceda4328c54df" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } } peers { permanent_uuid: "6a389d02268e457b8da60f9bf1329ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } }
I20260924 03:15:01.020682 23425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a389d02268e457b8da60f9bf1329ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a389d02268e457b8da60f9bf1329ebe, State: Initialized, Role: FOLLOWER
I20260924 03:15:01.021286 23425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a389d02268e457b8da60f9bf1329ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0308f5c867b54d23a49cba5f2a8bb377" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } } peers { permanent_uuid: "2f41246ed4c04f04827ceda4328c54df" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } } peers { permanent_uuid: "6a389d02268e457b8da60f9bf1329ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } }
W20260924 03:15:01.022059 23446 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:15:01.022254 23446 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:15:01.010020 23209 master_runner.cc:186] Error getting master registration for 127.0.97.61:38763: OK, instance_id { permanent_uuid: "2f41246ed4c04f04827ceda4328c54df" instance_seqno: 1790219700365803 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:15:01.028699 23209 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:15:01.033846 23451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a389d02268e457b8da60f9bf1329ebe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0308f5c867b54d23a49cba5f2a8bb377" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } } peers { permanent_uuid: "2f41246ed4c04f04827ceda4328c54df" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } } peers { permanent_uuid: "6a389d02268e457b8da60f9bf1329ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } } }
I20260924 03:15:01.034482 23451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a389d02268e457b8da60f9bf1329ebe [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:15:01.035343 23425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a389d02268e457b8da60f9bf1329ebe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:15:01.035390 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0308f5c867b54d23a49cba5f2a8bb377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6a389d02268e457b8da60f9bf1329ebe"
I20260924 03:15:01.035934 23400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a389d02268e457b8da60f9bf1329ebe [term 0 FOLLOWER]: Advancing to term 1
W20260924 03:15:01.040876 23230 tablet.cc:2369] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:15:01.042397 23400 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6a389d02268e457b8da60f9bf1329ebe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0308f5c867b54d23a49cba5f2a8bb377 in term 1.
I20260924 03:15:01.043567 23225 leader_election.cc:302] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0308f5c867b54d23a49cba5f2a8bb377, 6a389d02268e457b8da60f9bf1329ebe; no voters:
I20260924 03:15:01.044160 23434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:01.060292 23434 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [term 1 LEADER]: Becoming Leader. State: Replica: 0308f5c867b54d23a49cba5f2a8bb377, State: Running, Role: LEADER
I20260924 03:15:01.061074 23434 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0308f5c867b54d23a49cba5f2a8bb377" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } } peers { permanent_uuid: "2f41246ed4c04f04827ceda4328c54df" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } } peers { permanent_uuid: "6a389d02268e457b8da60f9bf1329ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } }
I20260924 03:15:01.062870 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:01.029840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38318 (local address 127.0.97.62:34675)
0924 03:15:01.030049 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.030056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.037710 (+ 7654us) server_negotiation.cc:407] Connection header received
0924 03:15:01.038350 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.038361 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.038480 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.038579 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:01.041582 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.042684 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.046207 (+ 3523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.047198 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.047337 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:01.053321 (+ 5984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:01.053328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:01.053332 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:01.053384 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:01.057664 (+ 4280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:01.058572 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:01.058578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:01.058581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:01.058912 (+ 331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:01.059613 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:01.059618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:01.059648 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:01.060333 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:01.060811 (+ 478us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:01.061767 (+ 956us) server_negotiation.cc:299] Negotiation successful
0924 03:15:01.062196 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 03:15:01.075412 23457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0308f5c867b54d23a49cba5f2a8bb377. Latest consensus state: current_term: 1 leader_uuid: "0308f5c867b54d23a49cba5f2a8bb377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0308f5c867b54d23a49cba5f2a8bb377" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } } peers { permanent_uuid: "2f41246ed4c04f04827ceda4328c54df" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } } peers { permanent_uuid: "6a389d02268e457b8da60f9bf1329ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } } }
I20260924 03:15:01.076537 23457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:01.079334 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:01.043024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.0.97.60:37871)
0924 03:15:01.043207 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.043213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.053567 (+ 10354us) server_negotiation.cc:407] Connection header received
0924 03:15:01.053869 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.053877 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.053967 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.054042 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:01.058572 (+ 4530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.059783 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.061674 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.062325 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.062434 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:01.069485 (+ 7051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:01.069492 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:01.069497 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:01.069566 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:01.073213 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:01.076184 (+ 2971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:01.076192 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:01.076195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:01.076378 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:01.076928 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:01.076933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:01.076937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:01.077383 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:01.077732 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:01.078429 (+ 697us) server_negotiation.cc:299] Negotiation successful
0924 03:15:01.078781 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:15:01.082924 23463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:15:01.089913 23463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:15:01.092435 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:01.029349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51584 (local address 127.0.97.61:38763)
0924 03:15:01.029622 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.029629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.041787 (+ 12158us) server_negotiation.cc:407] Connection header received
0924 03:15:01.042683 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.042690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.042782 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.042856 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:01.063788 (+ 20932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.064963 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.066872 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.067627 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.067774 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:01.082316 (+ 14542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:01.082324 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:01.082329 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:01.082378 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:01.086601 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:01.088211 (+ 1610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:01.088218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:01.088222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:01.088525 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:01.089442 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:01.089451 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:01.089456 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:01.090118 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:01.090495 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:01.091505 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0924 03:15:01.091907 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 03:15:01.105607 23350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } attempt: 5
I20260924 03:15:01.106094 23400 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6a389d02268e457b8da60f9bf1329ebe [term 1 FOLLOWER]: Refusing update from remote peer 0308f5c867b54d23a49cba5f2a8bb377: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:01.107637 23457 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a389d02268e457b8da60f9bf1329ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 24 03:15:01 dist-test-slave-5l98 krb5kdc[23196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 03:15:01.138029 23451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a389d02268e457b8da60f9bf1329ebe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0308f5c867b54d23a49cba5f2a8bb377. Latest consensus state: current_term: 1 leader_uuid: "0308f5c867b54d23a49cba5f2a8bb377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0308f5c867b54d23a49cba5f2a8bb377" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } } peers { permanent_uuid: "2f41246ed4c04f04827ceda4328c54df" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } } peers { permanent_uuid: "6a389d02268e457b8da60f9bf1329ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } } }
I20260924 03:15:01.138417 23451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a389d02268e457b8da60f9bf1329ebe [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:15:01.140627 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:01.106168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37871 (local address 127.0.0.1:37796)
0924 03:15:01.106767 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:01.106788 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:01.106896 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:01.108063 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:01.108069 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:01.108630 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:01.108865 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:01.108882 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.111127 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.111133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:01.112587 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:01.112603 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.114824 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.114833 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:01.115120 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:01.116242 (+ 1122us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:01.116309 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:01.125017 (+ 8708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:01.136454 (+ 11437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:01.136464 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:01.136471 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:01.137176 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:01.138267 (+ 1091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:01.138273 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:01.138278 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:01.138504 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:01.139476 (+ 972us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:01.139482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:01.139678 (+ 196us) client_negotiation.cc:769] Sending connection context
0924 03:15:01.139854 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 03:15:01.140187 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478}
W20260924 03:15:01.140895 23379 tablet.cc:2369] T 00000000000000000000000000000000 P 6a389d02268e457b8da60f9bf1329ebe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:15:01.140937 23472 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 03:15:01.146855 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:01.106890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.0.97.60:37871)
0924 03:15:01.107153 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.107161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.107183 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:15:01.107236 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.107240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.107323 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.107426 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:01.109096 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.110208 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.113051 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.114015 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.114180 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:01.125270 (+ 11090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:01.125277 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:01.125282 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:01.125324 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:01.130860 (+ 5536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:01.137545 (+ 6685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:01.137553 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:01.137558 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:01.138026 (+ 468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:01.138810 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:01.138815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:01.138819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:01.139260 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:01.139567 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:01.146132 (+ 6565us) server_negotiation.cc:299] Negotiation successful
0924 03:15:01.146450 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260924 03:15:01.148926 23434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0308f5c867b54d23a49cba5f2a8bb377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0308f5c867b54d23a49cba5f2a8bb377" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } } peers { permanent_uuid: "2f41246ed4c04f04827ceda4328c54df" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } } peers { permanent_uuid: "6a389d02268e457b8da60f9bf1329ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } } }
I20260924 03:15:01.149286 23434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:01.151188 23463 catalog_manager.cc:1347] Generated new cluster ID: 81c3b2e1fe0e4fb5a04225be9934a290
I20260924 03:15:01.151279 23463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:15:01.149962 23451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a389d02268e457b8da60f9bf1329ebe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0308f5c867b54d23a49cba5f2a8bb377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0308f5c867b54d23a49cba5f2a8bb377" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } } peers { permanent_uuid: "2f41246ed4c04f04827ceda4328c54df" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } } peers { permanent_uuid: "6a389d02268e457b8da60f9bf1329ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } } }
I20260924 03:15:01.154320 23451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a389d02268e457b8da60f9bf1329ebe [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:15:01.157721 23350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f41246ed4c04f04827ceda4328c54df: Bootstrap starting.
I20260924 03:15:01.162130 23350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f41246ed4c04f04827ceda4328c54df: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:01.163568 23350 log.cc:824] T 00000000000000000000000000000000 P 2f41246ed4c04f04827ceda4328c54df: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:01.158857 23358 master_runner.cc:186] Error getting master registration for 127.0.97.61:38763: OK, instance_id { permanent_uuid: "2f41246ed4c04f04827ceda4328c54df" instance_seqno: 1790219700365803 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:15:01.177012 23358 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:15:01.180362 23350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f41246ed4c04f04827ceda4328c54df: No bootstrap required, opened a new log
I20260924 03:15:01.187187 23463 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:15:01.190088 23463 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:15:01.195523 23350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f41246ed4c04f04827ceda4328c54df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0308f5c867b54d23a49cba5f2a8bb377" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } } peers { permanent_uuid: "2f41246ed4c04f04827ceda4328c54df" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } } peers { permanent_uuid: "6a389d02268e457b8da60f9bf1329ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } }
I20260924 03:15:01.196110 23350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f41246ed4c04f04827ceda4328c54df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f41246ed4c04f04827ceda4328c54df, State: Initialized, Role: FOLLOWER
I20260924 03:15:01.196782 23350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f41246ed4c04f04827ceda4328c54df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0308f5c867b54d23a49cba5f2a8bb377" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } } peers { permanent_uuid: "2f41246ed4c04f04827ceda4328c54df" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } } peers { permanent_uuid: "6a389d02268e457b8da60f9bf1329ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } }
I20260924 03:15:01.198333 23476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f41246ed4c04f04827ceda4328c54df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0308f5c867b54d23a49cba5f2a8bb377" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } } peers { permanent_uuid: "2f41246ed4c04f04827ceda4328c54df" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } } peers { permanent_uuid: "6a389d02268e457b8da60f9bf1329ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } } }
I20260924 03:15:01.199215 23476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f41246ed4c04f04827ceda4328c54df [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:15:01.199780 23350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f41246ed4c04f04827ceda4328c54df [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:15:01.199818 23325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0308f5c867b54d23a49cba5f2a8bb377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2f41246ed4c04f04827ceda4328c54df"
I20260924 03:15:01.200301 23325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f41246ed4c04f04827ceda4328c54df [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:01.212548 23325 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2f41246ed4c04f04827ceda4328c54df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0308f5c867b54d23a49cba5f2a8bb377 in term 1.
I20260924 03:15:01.213255 23463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377: Generated new TSK 0
I20260924 03:15:01.213471 23324 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f41246ed4c04f04827ceda4328c54df [term 1 FOLLOWER]: Refusing update from remote peer 0308f5c867b54d23a49cba5f2a8bb377: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:01.214725 23463 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:15:01.215333 23457 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f41246ed4c04f04827ceda4328c54df" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:15:01.249316 23476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f41246ed4c04f04827ceda4328c54df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0308f5c867b54d23a49cba5f2a8bb377. Latest consensus state: current_term: 1 leader_uuid: "0308f5c867b54d23a49cba5f2a8bb377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0308f5c867b54d23a49cba5f2a8bb377" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } } peers { permanent_uuid: "2f41246ed4c04f04827ceda4328c54df" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } } peers { permanent_uuid: "6a389d02268e457b8da60f9bf1329ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } } }
I20260924 03:15:01.249917 23476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f41246ed4c04f04827ceda4328c54df [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:15:01.250447 23482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f41246ed4c04f04827ceda4328c54df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0308f5c867b54d23a49cba5f2a8bb377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0308f5c867b54d23a49cba5f2a8bb377" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } } peers { permanent_uuid: "2f41246ed4c04f04827ceda4328c54df" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } } peers { permanent_uuid: "6a389d02268e457b8da60f9bf1329ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } } }
I20260924 03:15:01.251147 23482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f41246ed4c04f04827ceda4328c54df [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:15:01.284385 23493 catalog_manager.cc:1259] Loaded cluster ID: 81c3b2e1fe0e4fb5a04225be9934a290
I20260924 03:15:01.284605 23493 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2f41246ed4c04f04827ceda4328c54df: loading cluster ID for follower catalog manager: success
I20260924 03:15:01.288347 23493 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2f41246ed4c04f04827ceda4328c54df: acquiring CA information for follower catalog manager: success
I20260924 03:15:01.291080 23493 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2f41246ed4c04f04827ceda4328c54df: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 03:15:01.329779 23431 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:15:01.311215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38334 (local address 127.0.97.62:34675)
0924 03:15:01.311478 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.311488 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.326016 (+ 14528us) server_negotiation.cc:407] Connection header received
0924 03:15:01.327757 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.327766 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.327874 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.327969 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:01.329383 (+ 1414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38334: BlockingRecv error: recv got EOF from 127.0.0.1:38334 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260924 03:15:01.330212 23432 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:15:01.310899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.0.97.61:38763)
0924 03:15:01.311102 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.311108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.328139 (+ 17031us) server_negotiation.cc:407] Connection header received
0924 03:15:01.328405 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.328414 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.328570 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.328656 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:01.329923 (+ 1267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51596: BlockingRecv error: recv got EOF from 127.0.0.1:51596 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260924 03:15:01.333478 23428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:15:01.326805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37800 (local address 127.0.97.60:37871)
0924 03:15:01.327589 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.327599 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.331236 (+ 3637us) server_negotiation.cc:407] Connection header received
0924 03:15:01.331528 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.331538 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.331657 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.331774 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:01.333160 (+ 1386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37800: BlockingRecv error: recv got EOF from 127.0.0.1:37800 (error 108)
Metrics: {"server-negotiator.queue_time_us":582}
I20260924 03:15:01.350354 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:01.342846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51600 (local address 127.0.97.61:38763)
0924 03:15:01.343059 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.343066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.343434 (+ 368us) server_negotiation.cc:407] Connection header received
0924 03:15:01.343718 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.343723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.343902 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.344105 (+ 203us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:01.344716 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.345604 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.347245 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.347887 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.348043 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:01.348819 (+ 776us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:01.349501 (+ 682us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:01.349700 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:01.349792 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 03:15:01.349889 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260924 03:15:01.368528 23428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:15:01.366018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.0.97.60:37871)
0924 03:15:01.366314 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.366325 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.366879 (+ 554us) server_negotiation.cc:407] Connection header received
0924 03:15:01.367177 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.367181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.367288 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.367425 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:01.368249 (+ 824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37812: BlockingRecv error: recv got EOF from 127.0.0.1:37812 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260924 03:15:01.423106 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:01.410611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51610 (local address 127.0.97.61:38763)
0924 03:15:01.410830 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.410840 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.417255 (+ 6415us) server_negotiation.cc:407] Connection header received
0924 03:15:01.417502 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.417513 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.417587 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.417698 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:01.418173 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.419261 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.420966 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.421513 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.421727 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:01.422182 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:01.422326 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:01.422449 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:01.422611 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 03:15:01.422720 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 03:15:01.439507 23428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:15:01.434727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.0.97.60:37871)
0924 03:15:01.434997 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.435008 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.437335 (+ 2327us) server_negotiation.cc:407] Connection header received
0924 03:15:01.437676 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.437688 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.437778 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.437915 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:01.439129 (+ 1214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37822: BlockingRecv error: recv got EOF from 127.0.0.1:37822 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 03:15:01.451843 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:01.444449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.0.97.61:38763)
0924 03:15:01.444751 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.444761 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.445329 (+ 568us) server_negotiation.cc:407] Connection header received
0924 03:15:01.445559 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.445564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.445647 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.445756 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:01.446208 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.447159 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.448842 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.449296 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.449452 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:01.449893 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:01.450015 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:01.450132 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:01.450235 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 03:15:01.450314 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260924 03:15:01.467325 23428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:15:01.462221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37838 (local address 127.0.97.60:37871)
0924 03:15:01.462464 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.462473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.465739 (+ 3266us) server_negotiation.cc:407] Connection header received
0924 03:15:01.465949 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.465955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.466045 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.466169 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:01.467072 (+ 903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37838: BlockingRecv error: recv got EOF from 127.0.0.1:37838 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 03:15:01.480768 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:01.474772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51630 (local address 127.0.97.61:38763)
0924 03:15:01.475089 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.475098 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.475618 (+ 520us) server_negotiation.cc:407] Connection header received
0924 03:15:01.475858 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.475864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.475950 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.476064 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:01.476526 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.477375 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.478891 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.479333 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.479483 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:01.479889 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:01.480034 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:01.480146 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:01.480307 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 03:15:01.480389 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
W20260924 03:15:01.498116 23428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:15:01.490610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37842 (local address 127.0.97.60:37871)
0924 03:15:01.490946 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.490956 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.496390 (+ 5434us) server_negotiation.cc:407] Connection header received
0924 03:15:01.496643 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.496650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.496748 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.496874 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:01.497873 (+ 999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37842: BlockingRecv error: recv got EOF from 127.0.0.1:37842 (error 108)
Metrics: {"server-negotiator.queue_time_us":162}
I20260924 03:15:01.517328 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:01.510610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51638 (local address 127.0.97.61:38763)
0924 03:15:01.510906 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.510916 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.511514 (+ 598us) server_negotiation.cc:407] Connection header received
0924 03:15:01.511801 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.511806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.511911 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.512043 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:01.512586 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.513641 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.515172 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.515694 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.515870 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:01.516338 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:01.516448 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:01.516652 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:01.516828 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 03:15:01.516920 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260924 03:15:01.526233 23428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:15:01.523718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37858 (local address 127.0.97.60:37871)
0924 03:15:01.524053 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.524063 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.524696 (+ 633us) server_negotiation.cc:407] Connection header received
0924 03:15:01.524934 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.524941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.525036 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.525170 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:01.525962 (+ 792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37858: BlockingRecv error: recv got EOF from 127.0.0.1:37858 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:15:01.551995 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:01.545938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.0.97.61:38763)
0924 03:15:01.546208 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.546216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.546768 (+ 552us) server_negotiation.cc:407] Connection header received
0924 03:15:01.547006 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.547011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.547086 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.547200 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:01.547617 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.548595 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.550132 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.550581 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.550737 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:01.551157 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:01.551276 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:01.551403 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:01.551562 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 03:15:01.551650 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 03:15:01.560652 23428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:15:01.558845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37860 (local address 127.0.97.60:37871)
0924 03:15:01.559031 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.559037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.559068 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:15:01.559335 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.559339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.559413 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.559561 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:01.560354 (+ 793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37860: BlockingRecv error: recv got EOF from 127.0.0.1:37860 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:15:01.602953 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:01.596688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51666 (local address 127.0.97.61:38763)
0924 03:15:01.596961 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.596970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.597571 (+ 601us) server_negotiation.cc:407] Connection header received
0924 03:15:01.597818 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.597828 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.597904 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.598017 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:01.598474 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.599306 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.600906 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.601509 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.601678 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:01.602112 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:01.602303 (+ 191us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:01.602408 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:01.602535 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 03:15:01.602618 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260924 03:15:01.611507 23428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:15:01.609687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37876 (local address 127.0.97.60:37871)
0924 03:15:01.609877 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.609886 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.609913 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:15:01.610230 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.610235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.610332 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.610460 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:01.611244 (+ 784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37876: BlockingRecv error: recv got EOF from 127.0.0.1:37876 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:15:01.686481 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:01.679691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51670 (local address 127.0.97.61:38763)
0924 03:15:01.680001 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.680008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.680391 (+ 383us) server_negotiation.cc:407] Connection header received
0924 03:15:01.680595 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.680601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.680696 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.680813 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:01.681360 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.682497 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.684030 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.684801 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.684959 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:01.685395 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:01.685564 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:01.685660 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:01.686016 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 03:15:01.686110 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260924 03:15:01.695031 23428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:15:01.692985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37878 (local address 127.0.97.60:37871)
0924 03:15:01.693212 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.693219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.693614 (+ 395us) server_negotiation.cc:407] Connection header received
0924 03:15:01.693784 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.693791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.693894 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.694020 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:01.694796 (+ 776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37878: BlockingRecv error: recv got EOF from 127.0.0.1:37878 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 03:15:01.736207 23457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0308f5c867b54d23a49cba5f2a8bb377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0308f5c867b54d23a49cba5f2a8bb377" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } } peers { permanent_uuid: "2f41246ed4c04f04827ceda4328c54df" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38763 } } peers { permanent_uuid: "6a389d02268e457b8da60f9bf1329ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37871 } } }
I20260924 03:15:01.736575 23457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0308f5c867b54d23a49cba5f2a8bb377 [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:01.832314 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:01.826974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51678 (local address 127.0.97.61:38763)
0924 03:15:01.827259 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.827265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.827529 (+ 264us) server_negotiation.cc:407] Connection header received
0924 03:15:01.827774 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.827780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.827856 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.827965 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:01.828440 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.829181 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.830555 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:01.830980 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:01.831126 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:01.831566 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:01.831677 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:01.831783 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:01.831961 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 03:15:01.832044 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260924 03:15:01.840765 23428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:15:01.838981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37888 (local address 127.0.97.60:37871)
0924 03:15:01.839238 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:01.839243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:01.839391 (+ 148us) server_negotiation.cc:407] Connection header received
0924 03:15:01.839555 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:01.839559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:01.839636 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:01.839729 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:01.840534 (+ 805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37888: BlockingRecv error: recv got EOF from 127.0.0.1:37888 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 03:15:02.107167 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:02.101488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51684 (local address 127.0.97.61:38763)
0924 03:15:02.101724 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:02.101737 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:02.101989 (+ 252us) server_negotiation.cc:407] Connection header received
0924 03:15:02.102225 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:02.102231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:02.102303 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:02.102411 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:02.102838 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:02.103557 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:02.105463 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:02.105877 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:02.106030 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:02.106430 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:02.106538 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:02.106639 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:02.106794 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 03:15:02.106878 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 03:15:02.115381 23428 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:15:02.113485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37890 (local address 127.0.97.60:37871)
0924 03:15:02.113685 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:02.113692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:02.113859 (+ 167us) server_negotiation.cc:407] Connection header received
0924 03:15:02.114067 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:02.114072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:02.114156 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:02.114268 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:02.115154 (+ 886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37890: BlockingRecv error: recv got EOF from 127.0.0.1:37890 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 03:15:02.145658 23472 catalog_manager.cc:1259] Loaded cluster ID: 81c3b2e1fe0e4fb5a04225be9934a290
I20260924 03:15:02.145890 23472 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6a389d02268e457b8da60f9bf1329ebe: loading cluster ID for follower catalog manager: success
I20260924 03:15:02.149034 23472 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6a389d02268e457b8da60f9bf1329ebe: acquiring CA information for follower catalog manager: success
I20260924 03:15:02.151000 23472 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6a389d02268e457b8da60f9bf1329ebe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:15:02.636627 23612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:02.631137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51694 (local address 127.0.97.61:38763)
0924 03:15:02.631770 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:02.631774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:02.631794 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:15:02.631967 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:02.631971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:02.632039 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:02.632127 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:02.632479 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:02.633325 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:02.634677 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:02.635059 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:02.635214 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:02.635731 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:02.635858 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:02.635933 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:02.636088 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 03:15:02.636157 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":416,"threads_started":1}
I20260924 03:15:02.649813 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:02.643230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37892 (local address 127.0.97.60:37871)
0924 03:15:02.643805 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:02.643810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:02.643828 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:15:02.643994 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:02.643999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:02.644244 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:02.644359 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:02.644983 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:02.645750 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:02.647466 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:02.647921 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:02.648018 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:02.648658 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:02.649236 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:02.649351 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:02.649471 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 03:15:02.649553 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":299,"threads_started":1}
I20260924 03:15:02.667018 23612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:02.656471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51702 (local address 127.0.97.61:38763)
0924 03:15:02.656695 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:02.656703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:02.661158 (+ 4455us) server_negotiation.cc:407] Connection header received
0924 03:15:02.661375 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:02.661384 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:02.661515 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:02.661630 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:02.662352 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:02.663074 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:02.665203 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:02.665629 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:02.665807 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:02.666291 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:02.666397 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:02.666504 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:02.666629 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 03:15:02.666715 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:15:02.669248 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:02.657483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37896 (local address 127.0.97.60:37871)
0924 03:15:02.657694 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:02.657702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:02.661893 (+ 4191us) server_negotiation.cc:407] Connection header received
0924 03:15:02.662057 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:02.662066 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:02.662203 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:02.662329 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:02.663220 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:02.664342 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:02.667346 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:02.667896 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:02.668041 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:02.668514 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:02.668658 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:02.668768 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:02.668884 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 03:15:02.668961 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:15:02.676782 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:02.656433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38348 (local address 127.0.97.62:34675)
0924 03:15:02.671657 (+ 15224us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:02.671662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:02.671679 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:15:02.671778 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:02.671781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:02.671945 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:02.672100 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:02.672673 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:02.673393 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:02.674829 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:02.675216 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:02.675303 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:02.675765 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:02.676096 (+ 331us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:02.676199 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:02.676353 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 03:15:02.676419 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15099,"thread_start_us":15715,"threads_started":1}
I20260924 03:15:02.682313 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 23209
I20260924 03:15:02.683355 23271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:02.796597 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 23209
I20260924 03:15:02.818240 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 23283
I20260924 03:15:02.819547 23345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:02.931569 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 23283
I20260924 03:15:02.952148 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu pid 23358
I20260924 03:15:02.953521 23420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:03.063762 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGuLdJ/build/asan/bin/kudu with pid 23358
2026-09-24T03:15:03Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5146 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9649 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (375860 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST